New section 4 Templates for large ensembles

[sebvi]

Initial request

In the context of the emerging machine learning based forecasting, it is not unusual to run probabilistic forecasting system of thousands or tens of thousands ensemble members due to the relative cheap computation costs of each member.

As of today, the maximum ensemble size in GRIB2 in Templates 4.1 and 4.11 is 255 because both the ensemble number and total ensemble size are encoded on 1 octet each only. We are proposing to create 2 new templates to increase significantly this limit by encoding both on 4 octets each instead.
The 2 following templates replicate templates 4.1 and 4.11 allocating more octets for the ensemble number and the ensemble size

Amendment details

ADD new entries in Code Table 4.0

Octet	Meaning
117	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time
118	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

ADD template 4.117 - individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time

Title_en	OctetNo	Contents_en	Note_en	noteIDs	codeTable	flagTable	Status
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	10	Parameter category	(see Code table 4.1)		4.1
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	11	Parameter number	(see Code table 4.2)		4.2
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	12	Type of generating process	(see Code table 4.3)		4.3
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	13	Background generating process identifier (defined by originating centre)
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	14	Forecast generating process identifier (defined by originating centre)
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	15-16	Hours after reference time of data cut-off	(see Note)	33
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	17	Minutes after reference time of data cut-off
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	18	Indicator of unit of time range	(see Code table 4.4)		4.4
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	19-22	Forecast time in units defined by octet 18
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	23	Type of first fixed surface	(see Code table 4.5)		4.5
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	24	Scale factor of first fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	25-28	Scaled value of first fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	29	Type of second fixed surface	(see Code table 4.5)		4.5
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	30	Scale factor of second fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	31-34	Scaled value of second fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	35	Type of ensemble forecast	(see Code table 4.6)		4.6
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	36-39	Perturbation number
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a point in time	40-43	Number of forecasts in ensemble

ADD template 4.118 - individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Title_en	OctetNo	Contents_en	Note_en	noteIDs	codeTable	flagTable	Status
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	10	Parameter category	(see Code table 4.1)		4.1
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	11	Parameter number	(see Code table 4.2)		4.2
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	12	Type of generating process	(see Code table 4.3)		4.3
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	13	Background generating process identifier (defined by originating centre)
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	14	Forecast generating process identifier (defined by originating centre)
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	15-16	Hours after reference time of data cut-off	(see Note 1)	33
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	17	Minutes after reference time of data cut-off
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	18	Indicator of unit of time range	(see Code table 4.4)		4.4
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	19-22	Forecast time in units defined by octet 18	(see Note 2)	111
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	23	Type of first fixed surface	(see Code table 4.5)		4.5
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	24	Scale factor of first fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	25-28	Scaled value of first fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	29	Type of second fixed surface	(see Code table 4.5)		4.5
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	30	Scale factor of second fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	31-34	Scaled value of second fixed surface
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	35	Type of ensemble forecast	(see Code table 4.6)		4.6
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	36-39	Perturbation number
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	40-43	Number of forecasts in ensemble
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	44-45	Year of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	46	Month of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	47	Day of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	48	Hour of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	49	Minute of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	50	Second of end of overall time interval
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	51	n - number of time range specifications describing the time intervals used to calculate the statistically processed field
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	52-55	Total number of data values missing in statistical process
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval		56-67 Specification of the outermost (or only) time range over which statistical processing is done
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	56	Statistical process used to calculate the processed field from the field at each time increment during the time range	(see Code table 4.10)		4.10
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	57	Type of time increment between successive fields used in the statistical processing	(see Code table 4.11)		4.11
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	58	Indicator of unit of time for time range over which statistical processing is done	(see Code table 4.4)		4.4
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	59-62	Length of the time range over which statistical processing is done, in units defined by the previous octet
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	63	Indicator of unit of time for the increment between the successive fields used	(see Code table 4.4)		4.4
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	64-67	Time increment between successive fields, in units defined by the previous octet	(see Notes 3 and 4)	7,99
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval		68-nn These octets are included only if n > 1, where nn = 55 + 12 x n
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	68-79	As octets 56 to 67, next innermost step of processing
	individual large ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval	80-nn	Additional time range specifications, included in accordance with the value of n. Contents as octets 56 to 67, repeated as necessary

Comments

No response

Requestor(s)

Sebastien Villaume (ECMWF)

Stakeholder(s)

ECMWF

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, GRIB, section 4 template 4.117 (new)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB, section 4 template 4.118 (new)

Expected impact of change

MEDIUM

Collaborators

No response

References

No response

Validation

No response

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.6.7.June.2023 

@sebvi add samples and update branch;

[sebvi]

branch updated

[amilan17]

@sebvi  

Template 4.118 references note 99 in octetNo 64-67, but the note references different octets. Can we remove those references from the note? 

note 99,"The reference and forecast times are successively set to their initial values plus or minus the increment, as defined by the type of time increment (one of octets 51, 63, 75, ...). For all but the innermost (last) time range, the next inner range is then processed using these reference and forecast times as the initial reference and forecast times."

[sebvi]

I think we can remove the mention to octets from the note so that the note is more generic.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.13.July.2023 notes:

@sebvi add samples @SibylleK validate

[sebvi]

@sebvi  

Template 4.118 references note 99 in octetNo 64-67, but the note references different octets. Can we remove those references from the note? 

note 99,"The reference and forecast times are successively set to their initial values plus or minus the increment, as defined by the type of time increment (one of octets 51, 63, 75, ...). For all but the innermost (last) time range, the next inner range is then processed using these reference and forecast times as the initial reference and forecast times."

it looks like notes 95 to 110 are all identical except for the list of octets they reference....How do we proceed then? we should either continue with this practice or use a generic note for all these cases

[sebvi]

a version of ecCodes able to encode/decode templates 4.117 and 4.118 can be found here:eccodes

I am also uploading samples for these 2 templates:
large_eps.zip

[SibylleK]

I was able to read the entries in section 4 of the GRIB examples with a DWD GRIB reader software.
The entries were the same as in an output of eccodes grib_dump, in particular 123456 for the perturbation number and 1234567 for number of forecasts in ensemble in both examples.

This should be sufficient for the validation of this proposal.
From my point of view, the validation is done.

[amilan17]

@amilan17 update notes in FT branch