ValueError: key present and new value is different...

[alexamici]

@darrenleeweber has reported on #2 a problem that deserves it's own issue:

Thanks, also bumped into this while trying to read a GFS grib2 file, e.g.

import cfgrib
ds = cfgrib.Dataset.frompath('gfs_4_20110807_0000_000.grb2')
# snipped traceback
~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in enforce_unique_attributes(index, attributes_keys)
    113         values = index[key]
    114         if len(values) > 1:
--> 115             raise ValueError("multiple values for unique attribute %r: %r" % (key, values))
    116         if values and values[0] not in ('undef', 'unknown'):
    117             attributes['GRIB_' + key] = values[0]

ValueError: multiple values for unique attribute 'typeOfLevel': ['isobaricInhPa', 'tropopause', 'maxWind', 'isothermZero', 'unknown', 'potentialVorticity']

The work around seems to work, but hits another snag for this particular data example, i.e.

ds = cfgrib.Dataset.frompath('gfs_4_20110807_0000_000.grb2', filter_by_keys={'typeOfLevel': 'isobaricInhPa'})
# snipped traceback
~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in build_dataset_components(stream, encode_parameter, encode_time, encode_vertical, encode_geography, filter_by_keys)
    374         vars = collections.OrderedDict([(short_name, data_var)])
    375         vars.update(coord_vars)
--> 376         dict_merge(dimensions, dims)
    377         dict_merge(variables, vars)
    378     attributes = enforce_unique_attributes(index, GLOBAL_ATTRIBUTES_KEYS)

~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in dict_merge(master, update)
    353         else:
    354             raise ValueError("key present and new value is different: "
--> 355                              "key=%r value=%r new_value=%r" % (key, master[key], value))
    356 
    357 

ValueError: key present and new value is different: key='air_pressure' value=26 new_value=25

It's not easy to figure out if this is cfgrib or the data is not conforming.

[alexamici]

@darrenleeweber can you point me to a way to download a sample file? Thanks.

[dazza-codes]

LMK if you don't have permissions to read this S3 file, it's just public data from NOAA GFS

• https://s3.amazonaws.com/jupiter-reference-test/gfs/irene_2011/gfs_4_20110807_0000_000.grb2

[alexamici]

Clicking on the link I get an Access Denied XML, without much of an error message.

[dazza-codes]

Try https://s3-us-west-2.amazonaws.com/dweber.consulting.tmp/gfs_4_20110807_0000_000.grb2

[alexamici]

No luck, same error :(

[alexamici]

@darrenleeweber can you confirm that downloading for example from https://nomads.ncdc.noaa.gov/data/gfs4/201808/20180804/ would work?

[dazza-codes]

Hopefully, the permissions are fixed now, so try

$ pip install awscli
$ aws s3 cp s3://dweber.consulting.tmp/gfs_4_20110807_0000_000.grb2 ./
download: s3://dweber.consulting.tmp/gfs_4_20110807_0000_000.grb2 to ./gfs_4_20110807_0000_000.grb2

Sorry about that; please ping me so I can take that down.

[alexamici]

It worked this time (FYI I added --no-sign-request), you can take the file down. Thanks!

[alexamici]

As of version 0.9.0 cfgrib reports a warning instead of raising an exception, with enough information for the user to build an appropriate filter_by_keys in most cases.

For example:

>>> cfgrib.open_dataset('gfs_4_20180804_0000_000.grb2', filter_by_keys={'typeOfLevel': 'isobaricInhPa'})
skipping variable with paramId==260131 shortName='o3mr'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=17
skipping variable with paramId==3041 shortName='absv'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=26
skipping variable with paramId==135 shortName='w'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=21
skipping variable with paramId==260018 shortName='clwmr'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=21
<xarray.Dataset>
Dimensions:       (air_pressure: 31, latitude: 361, longitude: 720)
Coordinates:
    time          datetime64[ns] ...
    step          timedelta64[ns] ...
  * air_pressure  (air_pressure) float64 1e+03 975.0 950.0 925.0 ... 3.0 2.0 1.0
  * latitude      (latitude) float64 90.0 89.5 89.0 88.5 ... -89.0 -89.5 -90.0
  * longitude     (longitude) float64 0.0 0.5 1.0 1.5 ... 358.5 359.0 359.5
    valid_time    datetime64[ns] ...
Data variables:
    gh            (air_pressure, latitude, longitude) float32 ...
    t             (air_pressure, latitude, longitude) float32 ...
    r             (air_pressure, latitude, longitude) float32 ...
    u             (air_pressure, latitude, longitude) float32 ...
    v             (air_pressure, latitude, longitude) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    history:                 GRIB to CDM+CF via cfgrib-0.9.0/ecCodes-2.8.2 wi...
>>> cfgrib.open_dataset('gfs_4_20180804_0000_000.grb2', filter_by_keys={'shortName': 'o3mr'})
<xarray.Dataset>
Dimensions:       (air_pressure: 17, latitude: 361, longitude: 720)
Coordinates:
    time          datetime64[ns] ...
    step          timedelta64[ns] ...
  * air_pressure  (air_pressure) float64 400.0 350.0 300.0 250.0 ... 3.0 2.0 1.0
  * latitude      (latitude) float64 90.0 89.5 89.0 88.5 ... -89.0 -89.5 -90.0
  * longitude     (longitude) float64 0.0 0.5 1.0 1.5 ... 358.5 359.0 359.5
    valid_time    datetime64[ns] ...
Data variables:
    o3mr          (air_pressure, latitude, longitude) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    history:                 GRIB to CDM+CF via cfgrib-0.9.0/ecCodes-2.8.2 wi...

[alexamici]

Raising or ignoring the exception is now configurable via the errors option in backend_kwargs. The default is strict that keep the exception but can be set to ignore to just drop the offending hypecubes.