Missing v component in nam.t06z.awip3d00.tm00.grib2 after filter_by_keys

[alexamici]

I think I may have found a bug or at least something I do not understand in the implementation of the filter_by_keys argument.

Here is some output I receive when trying to open one of those NAM grib files:

>>> xr.open_dataset('nam.t06z.awip3d00.tm00.grib2',
                    engine='cfgrib',
                    backend_kwargs={
                        'filter_by_keys': {'typeOfLevel': 'isobaricInhPa'},
                        'errors': 'ignore'
                    })

skipping variable: paramId==3041 shortName='absv'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=7
skipping variable: paramId==1 shortName='strf'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 430, in build_dataset_components
    dict_merge(variables, vars)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=Variable(dimensions=('isobaricInhPa',), data=array([1000,  975,  950,  925,  900,  875,  850,  825,  800,  775,  750,
        725,  700,  675,  650,  625,  600,  575,  550,  525,  500,  475,
        450,  425,  400,  375,  350,  325,  300,  275,  250,  225,  200,
        175,  150,  125,  100,   75,   50])) new_value=Variable(dimensions=(), data=250)
skipping variable: paramId==3017 shortName='dpt'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=6
skipping variable: paramId==260022 shortName='mconv'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=2

<xarray.Dataset>
Dimensions:        (isobaricInhPa: 39, x: 185, y: 129)
Coordinates:
    time           datetime64[ns] ...
    step           timedelta64[ns] ...
  * isobaricInhPa  (isobaricInhPa) int64 1000 975 950 925 900 ... 125 100 75 50
    latitude       (y, x) float64 ...
    longitude      (y, x) float64 ...
    valid_time     datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
    gh             (isobaricInhPa, y, x) float32 ...
    t              (isobaricInhPa, y, x) float32 ...
    r              (isobaricInhPa, y, x) float32 ...
    q              (isobaricInhPa, y, x) float32 ...
    w              (isobaricInhPa, y, x) float32 ...
    u              (isobaricInhPa, y, x) float32 ...
    tke            (isobaricInhPa, y, x) float32 ...
    clwmr          (isobaricInhPa, y, x) float32 ...
    cice           (isobaricInhPa, y, x) float32 ...
    snmr           (isobaricInhPa, y, x) float32 ...
    strf           (y, x) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             US National Weather Service - NCEP 
    history:                 GRIB to CDM+CF via cfgrib-0.9.5.1/ecCodes-2.8.0 ...

You can see that it successfully returns a Dataset, but looking at the variables it returns, there is the U component of the wind but not the V component of the wind. I'm not sure why this is the case, since I've inspected the grib and find nothing apparent wrong with the v-winds. I've additionally tried this on other NAM gribs with similar results (even in your comment above on Sept. 30 this was the case).

I am unsure if this is an error on my part or if there is a way around this.
Thank you.

Originally posted by @bbonenfant in #2 (comment)

[alexamici]

@bbonenfant finally I'm on this and I can confirm it is indeed a bug. grib_ls confirms the v variable is in the file (so ecCodes C-API sees it) but I get no trace of if it using the Python bindings. Still investigating.

[alexamici]

The plot thickens, by using grib_ls I see the v messages all right:

$ grib_ls nam.t00z.awp21100.tm00.grib2
nam.t00z.awp21100.tm00.grib2
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
...
2            kwbc         20180917     fc           lambert      0            isobaricInhPa  100          w            grid_complex_spatial_differencing 
2            kwbc         20180917     fc           lambert      0            isobaricInhPa  100          u            grid_complex_spatial_differencing 
2            kwbc         20180917     fc           lambert      0            isobaricInhPa  100          v            grid_complex_spatial_differencing 
2            kwbc         20180917     fc           lambert      0            isobaricInhPa  150          gh           grid_complex_spatial_differencing 
...

But by looping on the lowest level python bindings all (and only) the v messages are skipped:

>>> ds = cfgrib.FileStream('nam.t00z.awp21100.tm00.grib2', errors='strinct')
>>> o = 0
... for i, msg in enumerate(ds):
...     print(i + 1, msg['shortName'], msg['offset'] - o)
...     o = msg['offset']
...
6 w 2398.0
7 u 8091.0
8 gh 13141.0
...

@shahramn I only use codes_handle_new_from_file that appear to NOT fail and just skip to the next message. Any idea?

[alexamici]

@shahramn identified that the nam GRIB files use MULTI-FIELD messages, a rare feature, that ecCodes can handle just fine but only after being requested to do it by calling codes_grib_multi_support_on.

I'll need a bit of time to implement the feature correctly, but it is basically solved.

[alexamici]

The bug was not really fixed as the FieldIndex gets confused by the multi-field messages having the same offset. More evident after fixing #54 via #58 .

[bbonenfant]

Hello again, I'm just now revisiting this, an I'm still experiencing the same bug where the u winds are present but the v winds are not. This is using cfgrib version 0.9.7.2. Below is the output from opening a NAM awphys** GRIB2 file which can be within this database. I've made sure to check all the array.Dataset objects output by the cfgrib.open_datasets method and none contain v winds.

<xarray.Dataset>
Dimensions:        (isobaricInhPa: 39, x: 614, y: 428)
Coordinates:
    time           datetime64[ns] 2019-10-30
    step           timedelta64[ns] 00:00:00
  * isobaricInhPa  (isobaricInhPa) int64 1000 975 950 925 900 ... 125 100 75 50
    latitude       (y, x) float64 12.19 12.22 12.25 12.28 ... 57.39 57.36 57.33
    longitude      (y, x) float64 226.5 226.6 226.8 226.9 ... 310.3 310.4 310.6
    valid_time     datetime64[ns] 2019-10-30
Dimensions without coordinates: x, y
Data variables:
    t              (isobaricInhPa, y, x) float32 ...
    u              (isobaricInhPa, y, x) float32 ...
    w              (isobaricInhPa, y, x) float32 ...
    gh             (isobaricInhPa, y, x) float32 ...
    r              (isobaricInhPa, y, x) float32 ...
    tke            (isobaricInhPa, y, x) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             US National Weather Service - NCEP