Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ValueError: key present and new value is different... #13

Closed
alexamici opened this Issue Aug 13, 2018 · 10 comments

Comments

Projects
None yet
2 participants
@alexamici
Copy link
Collaborator

alexamici commented Aug 13, 2018

@darrenleeweber has reported on #2 a problem that deserves it's own issue:

Thanks, also bumped into this while trying to read a GFS grib2 file, e.g.

import cfgrib
ds = cfgrib.Dataset.frompath('gfs_4_20110807_0000_000.grb2')
# snipped traceback
~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in enforce_unique_attributes(index, attributes_keys)
    113         values = index[key]
    114         if len(values) > 1:
--> 115             raise ValueError("multiple values for unique attribute %r: %r" % (key, values))
    116         if values and values[0] not in ('undef', 'unknown'):
    117             attributes['GRIB_' + key] = values[0]

ValueError: multiple values for unique attribute 'typeOfLevel': ['isobaricInhPa', 'tropopause', 'maxWind', 'isothermZero', 'unknown', 'potentialVorticity']

The work around seems to work, but hits another snag for this particular data example, i.e.

ds = cfgrib.Dataset.frompath('gfs_4_20110807_0000_000.grb2', filter_by_keys={'typeOfLevel': 'isobaricInhPa'})
# snipped traceback
~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in build_dataset_components(stream, encode_parameter, encode_time, encode_vertical, encode_geography, filter_by_keys)
    374         vars = collections.OrderedDict([(short_name, data_var)])
    375         vars.update(coord_vars)
--> 376         dict_merge(dimensions, dims)
    377         dict_merge(variables, vars)
    378     attributes = enforce_unique_attributes(index, GLOBAL_ATTRIBUTES_KEYS)

~/anaconda3/lib/python3.6/site-packages/cfgrib/dataset.py in dict_merge(master, update)
    353         else:
    354             raise ValueError("key present and new value is different: "
--> 355                              "key=%r value=%r new_value=%r" % (key, master[key], value))
    356 
    357 

ValueError: key present and new value is different: key='air_pressure' value=26 new_value=25

It's not easy to figure out if this is cfgrib or the data is not conforming.

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Aug 13, 2018

@darrenleeweber can you point me to a way to download a sample file? Thanks.

@alexamici alexamici self-assigned this Aug 13, 2018

@alexamici alexamici added bug enhancement and removed bug labels Aug 13, 2018

@darrenleeweber

This comment has been minimized.

Copy link

darrenleeweber commented Aug 13, 2018

LMK if you don't have permissions to read this S3 file, it's just public data from NOAA GFS

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Aug 13, 2018

Clicking on the link I get an Access Denied XML, without much of an error message.

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Aug 14, 2018

No luck, same error :(

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Aug 14, 2018

@darrenleeweber can you confirm that downloading for example from https://nomads.ncdc.noaa.gov/data/gfs4/201808/20180804/ would work?

@darrenleeweber

This comment has been minimized.

Copy link

darrenleeweber commented Aug 16, 2018

Hopefully, the permissions are fixed now, so try

$ pip install awscli
$ aws s3 cp s3://dweber.consulting.tmp/gfs_4_20110807_0000_000.grb2 ./
download: s3://dweber.consulting.tmp/gfs_4_20110807_0000_000.grb2 to ./gfs_4_20110807_0000_000.grb2

Sorry about that; please ping me so I can take that down.

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Aug 17, 2018

It worked this time (FYI I added --no-sign-request), you can take the file down. Thanks!

@alexamici alexamici added bug and removed enhancement labels Sep 15, 2018

@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Oct 14, 2018

As of version 0.9.0 cfgrib reports a warning instead of raising an exception, with enough information for the user to build an appropriate filter_by_keys in most cases.

For example:

>>> cfgrib.open_dataset('gfs_4_20180804_0000_000.grb2', filter_by_keys={'typeOfLevel': 'isobaricInhPa'})
skipping variable with paramId==260131 shortName='o3mr'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=17
skipping variable with paramId==3041 shortName='absv'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=26
skipping variable with paramId==135 shortName='w'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=21
skipping variable with paramId==260018 shortName='clwmr'
Traceback (most recent call last):
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 411, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/amici/devel/MPY/cfgrib/cfgrib/dataset.py", line 387, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='air_pressure' value=31 new_value=21
<xarray.Dataset>
Dimensions:       (air_pressure: 31, latitude: 361, longitude: 720)
Coordinates:
    time          datetime64[ns] ...
    step          timedelta64[ns] ...
  * air_pressure  (air_pressure) float64 1e+03 975.0 950.0 925.0 ... 3.0 2.0 1.0
  * latitude      (latitude) float64 90.0 89.5 89.0 88.5 ... -89.0 -89.5 -90.0
  * longitude     (longitude) float64 0.0 0.5 1.0 1.5 ... 358.5 359.0 359.5
    valid_time    datetime64[ns] ...
Data variables:
    gh            (air_pressure, latitude, longitude) float32 ...
    t             (air_pressure, latitude, longitude) float32 ...
    r             (air_pressure, latitude, longitude) float32 ...
    u             (air_pressure, latitude, longitude) float32 ...
    v             (air_pressure, latitude, longitude) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    history:                 GRIB to CDM+CF via cfgrib-0.9.0/ecCodes-2.8.2 wi...
>>> cfgrib.open_dataset('gfs_4_20180804_0000_000.grb2', filter_by_keys={'shortName': 'o3mr'})
<xarray.Dataset>
Dimensions:       (air_pressure: 17, latitude: 361, longitude: 720)
Coordinates:
    time          datetime64[ns] ...
    step          timedelta64[ns] ...
  * air_pressure  (air_pressure) float64 400.0 350.0 300.0 250.0 ... 3.0 2.0 1.0
  * latitude      (latitude) float64 90.0 89.5 89.0 88.5 ... -89.0 -89.5 -90.0
  * longitude     (longitude) float64 0.0 0.5 1.0 1.5 ... 358.5 359.0 359.5
    valid_time    datetime64[ns] ...
Data variables:
    o3mr          (air_pressure, latitude, longitude) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    history:                 GRIB to CDM+CF via cfgrib-0.9.0/ecCodes-2.8.2 wi...
@alexamici

This comment has been minimized.

Copy link
Collaborator Author

alexamici commented Nov 3, 2018

Raising or ignoring the exception is now configurable via the errors option in backend_kwargs. The default is strict that keep the exception but can be set to ignore to just drop the offending hypecubes.

@alexamici alexamici closed this in d3d4896 Nov 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.