Composite local time in GRIB message

[sebvi]

Branch
https://github.com/wmo-im/GRIB2/tree/issue-06

~
In the context of Fire Forecasting, we are creating composite GRIB messages of several UTC time.

Typically, Fire forecasting (risk of fire) is only relevant during day time. Usually, when encoding a GRIB message, the time is UTC and fixed for the entire grid domain (which is ok most of the time). For Fire forecasting however, if we were using universal time, then the data value would be relevant only for part of the horizontal domain. For instance, 12Z will be relevant for Europe but will be useless for Australia.

One could save several GRIB messages for several UTC time so that all continents are covered at some point with relevant, meaningful values. But it is much more practical to create composite GRIB messages that are composed by stripes of local times so that the data represents for instance the values for "noon" everywhere.

Practically, depending on the resolution of the model, we create GRIB files with 4 stripes (6 hour zones), 8 stripes (3 hours zones), 12 stripes (2 hours zones) or 24 stripes (hourly zones). The stripes are perfect longitudinal time zones (not following countries like the time zones we usually know).

The bottom part shows an example of 8 stripes (3 hours timestep) for a forecast starting at 00Z. each stripes in longitudes 360/8 degrees wide, i.e. 45 degrees: 337.5 to 22.5 degrees is 00z+12 UTC, 22.5 to 67.5 degrees is 00Z + 09 UTC, .... 292.5 to 337.5 degrees is 00Z+15 UTC

My feeling is that I need a whole new set of templates in section 4 that do not represent "point in time" or "continuous or non continuous time interval" but a set of keys to represent "composite time".

Alternatively an entry could be added in code table 1.2 to specify that the time in section 2 is still the reference time of the forecast (that still needs to be recorded) but that the time specified in section 4 will be a composite. The problem then is that I don't know how to specify the number of "stripes" which should also be recorded.

If anyone have suggestions, comments, alternatives?

[sebvi]

Please @wmo-im/tdcf have a look at this proposal :)

[sebvi]

@wmo-im/tdcf I have updated the issue with a picture and some extra explanations. Feedback, ideas are most welcome because we need to find the proper way to describe the metadata.

[ckpan-hko]

It's a good idea from user's perspective. But for GRIB products carrying global data, I think one single time domain would be better.

[efucile]

If I understand correctly there is a connection between time and grid. We can see it in different ways, but the connection is there. For different time steps we need to define a different slice of a global grid, connected with a different slice of data in the data section. This is very complex in GRIB and would require a careful implementation in the decoders. Not impossible, only very complex.

[sebvi]

Yes it is very complex! This is why I wanted an open discussion first before digging into this!

I could try to propose new sets of templates but it will be too short time to do so and too short notice to reasonably discuss it tomorrow. With the new github workflow, it seems much easier to me to consider approving requests more frequently than twice a year.

[ckpan-hko]

If we see "time" as a parameter with "UTC" as its base unit, does it mean that the "time" in different strips carries different unit?

[sebvi]

UPDATE

We continued to work on this proposal and here is where we are:

purpose of the proposal: Generate a template which allows to store an instantaneous field at a given local time using an appropriate forecast window to collate different UTC times

The simpler way to achieve this for each grid point and for a given solar/local time, is to:

1. calculate what is the exact UTC time at this location that corresponds to the chosen solar time
2. select grid point data (eg. Temperature) from a set of forecast step that is closest in UTC time to the exact calculated UTC time
   Following these simple rules, stripes of data are created and collated together. The better the time resolution of the input forecast, the narrower the stripes become and the less approximate is the solution. Basically We obtain schematically what is in the figures of the original post of this proposal.

However, this simple method generates artifacts as shown below with temperature where the strips are still visible (over Africa for instance).

A way to mitigate this is by interpolating data values to the given local time from the two closest UTC time steps as shown by the example below where a zoom is performed on one of the discontinuities.

The temperature field is interpolated in the plot on the right and is simply concatenated (stripes) in the plot on the left. If we look at two points, 1 and 2, very close to the discontinuity and plot the time serie of the 24 hours forecast, in the case of the concatenation, those points will be attributed temperature from two different time stamps. Instead in the case of the interpolation values from two nearest time stamps are interpolated leading to a more accurate result.

Proposal

1. add a new entry in Code Table 1.2 Significance of reference time: "4 - Local Time" with a note explaining that the grid points represents the value of the parameters for the local time encoded in the reference time of the message. "local Time" should be used together with specific products Definition Templates in section 4 designed to be used with it and specifying how the data points were obtained and from which forecast steps.

2. create a products definition template in section 4 that describes what are the forecasts steps used and what methods has been used to obtain the parameter value (nearest forecast step (stripes), time interpolation, etc.)

We are in the process of drafting the template.

[sebvi]

Proposal

add a new entry in Code table 1.2 Significance of reference time

Code	Meaning
4	Local Time

add in Code Table 4.0 an entry for the new template

Code	Meaning
88	Analysis or Forecast at a horizontal level or in a horizontal layer at a local Time

add a new Product definition template 4.88 - Analysis or Forecast at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12	Type of generating process (see Code table 4.3)
13	Background generating process identifier (defined by originating centre)
14	Forecast generating process identifier (defined by originating centre)
15	Type of first fixed surface (see Code table 4.5)
16	Scale factor of first fixed surface
17–20	Scaled value of first fixed surface
21	Type of second fixed surface (see Code table 4.5)
22	Scale factor of second fixed surface
23–26	Scaled value of second fixed surface
27	Method used to calculate the field value at the local time specified in section 1 (see Code table 4.248)
28	n - number of forecasts used in the processing
n/a	29-46 Specification of the first forecast used in the processing (reference date, reference time and forecast time increments)
29-30	Year of the reference date for the first forecast used in the processing
31	Month of the reference date for the first forecast used in the processing
32	Day of the reference date for the first forecast used in the processing
33	Hour of the reference time for the first forecast used in the processing
34	Minute of the reference time for the first forecast used in the processing
35	Second of the reference time for the first forecast used in the processing
36	Indicator of units of forecast time (see Code table 4.4)
37-40	Forecast time for the first forecast used in the processing, in units defined by the previous octet
41	Number of time increments for the first forecast used in the processing
42	Indicator of unit of time for the time increment between the successive forecast times used in the processing (see Code table 4.4)
43-46	Time increment between successive forecast times, in units defined by the previous octet
n/a	47-nn These octets are included only if n>1 where nn=28+18*n
47-nn	(n-1) repetitions of sequence of octet 29-46, describing the next forecasts used in the processing (reference date, reference time and forecast time increments)

create a new Code Table 4.248 Method used to derive the data values for a given local time

Code figure	Meaning
0	Nearest forecast time to local time
1	Interpolated to be valid at the specified local time
2-191	Reserved
192 - 254	Reserved for local use
255	Missing

[SimonElliottEUM]

Left hand column of Code table 4.248 above should be "Code figure" rather than "Octet". But otherwise this look plausible to me

[shahramn]

Octet 28 : "n - number of forecast used in the processing"
should read
"n - number of forecasts used in the processing"

[sebvi]

thank you @SimonElliottEUM and @shahramn , I have edited my proposal following your comments.

[sebvi]

I could not find a branch "issue-06" for this issue, any chance to create one?

[amilan17]

I could not find a branch "issue-06" for this issue, any chance to create one?

https://github.com/wmo-im/GRIB2/tree/issue-06

[sebvi]

@wmo-im/tt-tdcf : I would be grateful if others could comment on this proposal ahed of tomorrow's meeting. I will provide some sample files encoded with this template today.

During the final review of the proposal we realized that it would be useful to provide an additional template with the possibility to encode statistically processed fields. In the context of fire forecasting it is useful to provide the total precipitation during the 24 hours preceding the local time encoded. I will provide the template later today and would be extremely grateful if it can be reviewed before tomorrow's meeting.

[sebvi]

I tried to upload a sample for this template, not sure it worked properly as I don't see it anywhere

[jitsukoh]

@sebvi I don't see sample data. I'd appreciate anyone's help in validation but there is not decoder available, could you provide a decode output of ecCodes to complete the validation process? I understand that the use of products encoded using this new template will be limited to users of ecCodes at this moment and therefore there is no reason not to approve it because there are no other decoders available. Please comment if there is objection.

[sebvi]

Yes @jitsukoh I can definitely do that! I was hoping @amilan17 could find out why I can't upload files

[amilan17]

@sebvi -- I was able to upload this file after changing this box to support markdown. Can you try again? If it still doesn't work, then the file may be too large... (25 MB max)

[sebvi]

thank you Anna, I will upload tomorrow! I had a pretty busy week until now and could not find the time to do it Sebastien

…

On 21 Jan 2021, at 10:37, Anna Milan ***@***.***> wrote:  @sebvi -- I was able to upload this file after changing this box to support markdown. Can you try again? If it still doesn't work, then the file may be too large... (25 MB max) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[sebvi]

it does not seem to work for me... it worked before because I could attach pictures in this issue.

I will try to send as attachment in an email, maybe this trick will work

[sebvi]

here is a sample implementing PDT 88. I remove the bipmap and data section to reduce size
PDT88_ECMWF.zip

and here is a dump of the sample using ecCodes:
PDT88_ECMWF.txt