Reading WP-MIP grib2

[mpm-meto]

The WP-MIP data portal [https://hpfx.collab.science.gc.ca/~swpm001/selector.html] provides grib2 files which I want to put through METplus.

The grib files are monthly chunks for a single variable, all lead times & inits in the same file. You can download by variable/level. Here I am looking at T250.

Running wgrib2 shows this sort of thing. The files are in init with 6-hourly output to 240h with a t+0h [I am not going to vouch for this being the case for all models. It should be but I have looked at another model and found the time info different!] At least this is the case for the AIFS. [t_250_oic_ecmf_ai_00_pl_02.grib2]

62:60549961:d=2024020600:TMP:250 mb:anl:
63:61662334:d=2024020600:TMP:250 mb:6 hour fcst:
64:62715185:d=2024020600:TMP:250 mb:12 hour fcst:
65:63753165:d=2024020600:TMP:250 mb:18 hour fcst:
66:64780140:d=2024020600:TMP:250 mb:24 hour fcst:
67:65799667:d=2024020600:TMP:250 mb:30 hour fcst:
68:66813009:d=2024020600:TMP:250 mb:36 hour fcst:
69:67823323:d=2024020600:TMP:250 mb:42 hour fcst:
70:68829607:d=2024020600:TMP:250 mb:48 hour fcst:
71:69838750:d=2024020600:TMP:250 mb:54 hour fcst:
72:70842815:d=2024020600:TMP:250 mb:60 hour fcst:
73:71845614:d=2024020600:TMP:250 mb:66 hour fcst:
74:72846014:d=2024020600:TMP:250 mb:72 hour fcst:
75:73847940:d=2024020600:TMP:250 mb:78 hour fcst:
76:74847157:d=2024020600:TMP:250 mb:84 hour fcst:

That looks ok. Next, using the grib_dump utility to decode the octets for Section 1& 4 you get this:
====================== SECTION_1 ( length=21, padding=0 ) ======================
1-4 section1Length = 21
5 numberOfSection = 1
6-7 centre = 323 [Unknown code table entry (common/c-11.table) ]
8-9 subCentre = 98
10 tablesVersion = 30 [Version implemented on 15 November 2022 (grib2/tables/1.0.table) ]
11 localTablesVersion = 0 [Unknown code table entry () ]
12 significanceOfReferenceTime = 1 [Start of forecast (grib2/tables/30/1.2.table) ]
13-14 year = 2024
15 month = 2
16 day = 3
17 hour = 0
18 minute = 0
19 second = 0
20 productionStatusOfProcessedData = 17 [Unknown code table entry (grib2/tables/30/1.3.table) ]
21 typeOfProcessedData = 1 [Forecast products (grib2/tables/30/1.4.table) ]

====================== SECTION_4 ( length=34, padding=0 ) ======================
1-4 section4Length = 34
5 numberOfSection = 4
6-7 NV = 0
8-9 productDefinitionTemplateNumber = 0 [Analysis or forecast at a horizontal level or in a horizontal layer
at a point in time (grib2/tables/30/4.0.table) ]
10 parameterCategory = 0 [Temperature (grib2/tables/30/4.1.0.table) ]
11 parameterNumber = 0 [Temperature (K) (grib2/tables/30/4.2.0.0.table) ]
12 typeOfGeneratingProcess = 2 [Forecast (grib2/tables/30/4.3.table) ]
13 backgroundProcess = 1
14 generatingProcessIdentifier = 1
15-16 hoursAfterDataCutoff = 0
17 minutesAfterDataCutoff = 0
18 indicatorOfUnitForForecastTime = 1 [Hour (grib2/tables/30/4.4.table) ]
19-22 forecastTime = 0
23 typeOfFirstFixedSurface = 100 [Isobaric surface (Pa) (grib2/tables/30/4.5.table) ]
24 scaleFactorOfFirstFixedSurface = 0
25-28 scaledValueOfFirstFixedSurface = 25000
29 typeOfSecondFixedSurface = 255 [Missing (grib2/tables/30/4.5.table) ]
30 scaleFactorOfSecondFixedSurface = MISSING
31-34 scaledValueOfSecondFixedSurface = MISSING

That all looks ok... as far as my grib knowledge goes (which isn't far)

Then, on to the METplus output. METplus fails. Here are some lines from the log file.

getdrstemplate: DRS Template 5.42 not defined.

DEBUG 3: Multiple GRIB2 table entries match lookup criteria (parm_name = TMP):
DEBUG 3: parm_name: TMP, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 0, index_c = 0
DEBUG 3: parm_name: TMP, index_a = 0, grib2_mtab = 1, grib2_cntr = 7, grib2_ltab = 1, index_b = 0, index_c = 0
DEBUG 3: Using the first match found: parm_name: TMP, index_a = 0, grib2_mtab = 1, grib2_cntr = 0, grib2_ltab = 0, index_b = 0, index_c = 0

WARNING: Record 368 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 234, 100, 0, 25000, 255, -127, -2147483647
WARNING: Record 369 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 240, 100, 0, 25000, 255, -127, -2147483647
WARNING: Try setting the 0-based "GRIB2_ipdtmpl_index" and "GRIB2_ipdtmpl_val" config file options to select one.
WARNING:
FATAL ERROR (SEGFAULT): Process 1449050 got signal 11 @ local time = 2026-04-16 08:55:19Z

I have searched and searched but can’t quite make the link between the tables here https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-0.shtml and the MET documentation under 5.13.2.9 in https://metplus.readthedocs.io/projects/met/en/latest/Users_Guide/config_options.html.

How does one work out which index is which and what to set?

With the grib_dump octets I thought I figured out which index was the lead time but it wasn't right, and the output went from telling me it can't pick a unique record, to it can't find anything to match! I clearly didn't pick the right one.

How do i work out which ipdtmpl indices to set and what do I set them to? And what if, as I suspect, the grib table is not the same as MET uses, how does one get around that?

I know folks at the DTC will want to work with these files too so I am sure they may find this of interest [@KathrynNewman and @michelleharrold].

[RachelNorth]

So, I managed to pick out the correct index value to get the lead time (8), but it filters the number of fields down to 9 matching records.
WARNING: MetGrib2DataFile::data_plane_array() -> Using the first of 9 exact matching records for "TMP/P850":
WARNING: Record 5 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 46 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 87 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 128 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 169 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 210 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 251 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 292 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647
WARNING: Record 333 field 1, table 4.0: ipdtmpl[15] = 0, 0, 4, 255, 1, 0, 0, 1, 24, 100, 0, 85000, 255, -127, -2147483647

Having inspected the GRIB headers of these fields, the difference between them now is the date/time information.
Why is MET not selecting the correct time slice?

[mpm-meto]

Thanks @RachelNorth for digging even further! I really appreciate it. I think one of the things that has thrown us is the index in the print statement ipdtmpl[15]. It is really unclear what the "15" refers to. Or have you figured that out? It is likely completely irrelevant...

[RachelNorth]

I think the index in the print statement is simply stating that it's printing 15 items from the ipdtmpl. Each element of the print statement corresponds to one of the GRIB2 headers in that section (Section 4). So, number each one from left to right starting at 0 and that gives you the number of the index to use for that GRIB2 header in Section 4 of the message.

[RachelNorth]

Also, digging into the GRIB2 documentation at: https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/ suggests that Data Representation Template 5.42 is defined at https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp5-42.shtml.
How do we make sure these definitions are in our METplus builds?

[mpm-meto]

@RachelNorth our local copy of grib2 templates is here ~cfver/MET_Repos/MET_Dependencies/MET/MET-13.0.0-beta1/data/table_files but searching for 5.42 yields nothing. Whatever is needed (to begin with), is not being taken from here.

[JohnHalleyGotway]

@mpm-meto, thanks for pointing us to the WP-MIP Data Portal. I assume that WP-MIP is not actually generating any GRIB2 data, they're just redistributing data generated by other centers.

We should be able to configure METplus to account for how the GRIB2 records are organized into files, but it may require using some new options in METplus config files. So I'm less worried about that.

To test, I filled out this data selector site to generate this pull command:

wget -e robots=off -nd -r --no-parent --no-verbose --show-progress --rejected-log=/dev/null -A "\
t_250_oic_ecmf_ai_00_pl_02.grib2\
" http://hpfx.collab.science.gc.ca/~swpm001/WP-MIP/forecasts/oic/ecmf/ai_00/

And pulled this t_250_oic_ecmf_ai_00_pl_02.grib2 file. It contains 549 records of TMP/P250, each with different timing information.

• 9 different initialization times in February (20240203, 20240206, 20240209, 20240212, 20240215, 20240218, 20240221, 20240224, 20240227).
• 51 Forecast hours (0 to 360 every 6 hours).

Yes, you technically can distinguish between these by setting the idptmpl options, but that's meant as a last resort. It's much more convenient to just specify the timing info directly with the init_time, valid_time, or lead_time config options.

For example:

plot_data_plane t_250_oic_ecmf_ai_00_pl_02.grib2 plot.ps name="TMP"; level="P250";

Produces a lot of warnings like this:

WARNING: MetGrib2DataFile::data_plane() -> Using the first of 549 matching records for "TMP/P250":
WARNING:   Record 1 field 1, table 4.0: ipdtmpl[15] = 0, 0, 2, 1, 1, 0, 0, 1, 0, 100, 0, 25000, 255, -127, -2147483647
...

Whereas adding the init_time and lead_time options to select a single record:

plot_data_plane t_250_oic_ecmf_ai_00_pl_02.grib2 plot.ps \
'name="TMP"; level="P250"; init_time="20240203"; lead_time="120";'

Does not produce any warnings about multiple matching records.

HOWEVER, I do see other GRIB2 issues that warrant further investigation...

getdrstemplate: DRS Template 5.42 not defined.
FATAL ERROR (SEGFAULT): Process 2530905 got signal 11 @ local time = 2026-04-20 17:11:55Z
FATAL ERROR (SEGFAULT): Look for a core file in /d1/projects/METplus/discussions/3281

[JohnHalleyGotway]

@mpm-meto I did some more digging and confirmed the following:

1. We're currently using an older version (v1.6.4) of the GRIB2C library does NOT support AEC compression in our development and testing environment.
2. I downloaded a newer version of GRIB2C (v2.2.0) that does support AEC compression, installed upgraded versions of the JASPER and AEC libraries, and configured/compiled the GRIB2C library using -DUSE_AEC=YES.
3. After recompiling MET pointing to that GRIB2C v2.2.0 build with AEC support, I was able to read that ECMWF data without error.

For example:

plot_data_plane data/t_250_oic_ecmf_ai_00_pl_02.grib2 plot.ps \
'name="TMP"; level="P250"; init_time="20240203"; lead_time="120";'

Produces this plot:

While our development and testing environment does not currently support AEC compression, we do plan on updating the package versions for the next release. And I confirmed that doing so using the existing MET release does indeed work as you'd hope!

[mpm-meto]

Thanks @JohnHalleyGotway for digging. So, if I am reading this correctly, it doesn't need a new MET build but updating a dependency (and a local recompile?). Could you provide more specific instructions to @robdarvell please as he would be the one updating the dependencies as you have done? I think he may need a bit more.

[robdarvell]

Have downloaded the relevant GRIB2C version as described above. I think I just need to understand which version of the JASPER libraries I am trying to look for and if there are any nuiances in the build process compared to what is currently done (I did try 4.2.9 but that fell over and I haven't investigated as yet in case this isn't actually what I need). I have found all sorts of versions of the JASPER stuff, so best to get what value is necessary (as they didn't compare close to the version in the MET install).
I think we may be OK with the AEC bit if I have understood what we have correctly (but that maybe questionable)

[JohnHalleyGotway]

@mpm-meto, yes, I'd recommend recompiling the existing version of MET using updated versions of the GRIB2C library, Jasper, and AEC. I'll list the details of the versions I used and why, but other combinations of versions may work as well:

1. AEC library version 1.1.3 (since 1.1.5 and 1.1.6 require a newer version of cmake than what's currently on my Mac):

wget https://gitlab.dkrz.de/dkrz-sw/libaec/-/archive/v1.1.3/libaec-v1.1.3.tar.gz

2. Jasper library version 2.0.33 (needs to be > 2.0.25 but there certainly are a lot of newer versions available):

wget https://github.com/jasper-software/jasper/releases/download/version-2.0.33/jasper-2.0.33.tar.gz

3. GRIB2C library version 2.2.0 (since that's the version to which EMC is upgrading on WCOSS2):

wget https://github.com/NOAA-EMC/NCEPLIBS-g2c/archive/refs/tags/v2.2.0.tar.gz

Need to configure/compile GRIB2C using the -DUSE_AEC=YES option.

All three of these are compiled using cmake.

@robdarvell hope that helps!

[robdarvell]

Have downloaded and built the new versions of the 3 pieces of software, rebuilt the MET code base pointing to the new versions of GRIB2C and JASPER (there wasn't an option for AEC?), but rerunning the job is still complaining about 5.42 not defined.
That doesn't mean though that I haven't built things correctly mind you. I think the JASPER side is fine, it is more towards the AEC and GRIB2C linkage.
When doing the final install of the AEC software it is generating files such as libaec.so.0.1.3 rather than what I thought would be 1.1.3. The information in the include directory suggests it is the 1.1.3 version.
Anyway, when then pointing that to the GRIB2C install it originally talked about the version of AEC not being up to 1.0.6 (which was making me think it was still trying to use the generally installed version). By some jiggery pokery in the CMake file of GRIB2C I seemed to make it believe that the version was the correct one (and the include the lib64 pointed to my newly installed version of AEC) and that message went away, but no luck with the final build.

I will have another look next week, but if there are any places where I may have misinterpretted something then please do let me know.

[robdarvell]

@JohnHalleyGotway : Update on a Monday morning!!!
No idea what I did on Friday. Have rerun what I had from scratch again this morning, and it seems as though it has now worked. I have a plot which matches that which was posted above now. I believe Marion has also replicated using PlotDataPlane as well using my build.

[mpm-meto]

And update from me too... both plot_data_plane and PointStat wrapper with the updated build now work. But in the wrapper you definitely need this @RachelNorth FYI! Good detective work there :-) Thanks!
FCST_VAR1_OPTIONS = GRIB2_ipdtmpl_index=[8]; GRIB2_ipdtmpl_val = [{lead?fmt=%H}];

[JohnHalleyGotway]

Whew, glad to hear you got it sorted!

FYI, @jprestop is updating our standard builds of METplus for all of these Existing Builds for the next official release via issue dtcenter/MET#3377.

[mpm-meto]

As a final follow up to this... MET13.0.0Beta2 works!
It is worth point out though that in situations where ALL the forecast inits and lead times are in the same file, MET needs a bit more help to find a unique match for each init/lead. Like this:

FCST_VAR_OPTIONS = init_time = "{init?fmt=%Y%m%d}_00"; GRIB2_ipdtmpl_index=[8]; GRIB2_ipdtmpl_val = [{lead?fmt=%H}];
The use of "init_time" here is not explicitly stated anywhere but Copilot helped me find a random example in the user guide! (there is so much in there... you just need to find it!)

Now everything works a treat!

If you don't set this, it will give you a warning and always default to only looking at the very first init/lead it finds.
To be fair, this is a very strange way of storing forecasts but seems appropriate for the intercomparison. If the filenames had the init time in it, then this ambiguity would not arise.