Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dimension Co-ordinates incorectly saving _FillValue attribute #1865

Closed
czr137 opened this issue Jan 29, 2018 · 6 comments · Fixed by #1869
Closed

Dimension Co-ordinates incorectly saving _FillValue attribute #1865

czr137 opened this issue Jan 29, 2018 · 6 comments · Fixed by #1869

Comments

@czr137
Copy link
Contributor

czr137 commented Jan 29, 2018

Code Sample, a copy-pastable example if possible

import xarray as xr
import pandas as pd
import numpy as np

temp = 273.15 + 25 * np.random.randn(2, 2)
lon = [0.0, 5.0]
lat = [10.0, 20.0]

ds = xr.Dataset({'temperature': (['lat', 'lon'], temp)},
                coords={'lat': lat,
                        'lon': lon})

ds['lat'].attrs = {
    'standard_name': 'latitude',
    'long_name': 'latitude',
    'units': 'degrees_north',
    'axis': 'Y'}
ds['lon'].attrs = {
    'standard_name': 'longitude',
    'long_name': 'longitude',
    'units': 'degrees_east',
    'axis': 'X'}
ds['temperature'].attrs = {
    'standard_name': 'air_temperature',
    'units': 'K'}
ds.attrs = {
    ('title', 'non-conforming CF 1.6 data produced by xarray 0.10'),
    ('Conventions', 'CF-1.6')}

ds.to_netcdf('/tmp/test.nc')

Problem description

According to the last sentence of the first paragraph of 2.5.1. Missing data, valid and actual range of data in NetCDF Climate and Forecast (CF) Metadata Conventions 1.7:

Missing data is not allowed in coordinate variables.

When I use the conformance checker it issues an INFO message to this point for the co-ordinate variables.

Output of CF-Checker follows...

CHECKING` NetCDF FILE: /tmp/29428.nc
=====================
Using CF Checker Version 3.0.0
Checking against CF Version CF-1.6
Using Standard Name Table Version 48 (2017-11-28T15:32:48Z)
Using Area Type Table Version 6 (22 February 2017)


------------------
Checking variable: temperature
------------------

------------------
Checking variable: lat
------------------
INFO: attribute _FillValue is being used in a non-standard way

------------------
Checking variable: lon
------------------
INFO: attribute _FillValue is being used in a non-standard way

ERRORS detected: 0
WARNINGS given: 0
INFORMATION messages: 2

Expected Output

Co-ordinate variables should not store a _FillValue attribute

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.15-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.22.0
numpy: 1.14.0
scipy: 1.0.0
netCDF4: 1.3.1
h5netcdf: 0.5.0
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.1
matplotlib: 2.1.2
cartopy: None
seaborn: None
setuptools: 38.4.0
pip: 9.0.1
conda: None
pytest: None
IPython: None
sphinx: None

@czr137
Copy link
Contributor Author

czr137 commented Jan 29, 2018

Please add the label "CF conventions" to this issue

@shoyer
Copy link
Member

shoyer commented Jan 30, 2018

Please take a look at the discussion in #1598.

You can use ds.to_netcdf('/tmp/test.nc', encoding={'lat': {'_FillValue': None}, 'lon': {'_FillValue': None}}) to disable this.

@czr137
Copy link
Contributor Author

czr137 commented Jan 30, 2018

Thanks.

@czr137 czr137 closed this as completed Jan 30, 2018
@shoyer
Copy link
Member

shoyer commented Jan 30, 2018

If you have ideas for improving this further (even documentation), please do speak up.

@czr137 czr137 reopened this Jan 30, 2018
@czr137
Copy link
Contributor Author

czr137 commented Jan 30, 2018

My apologies for having to re-open the issue, I should have tried the fix before closing.

I get the following error after changing my last line to: ds.to_netcdf('/tmp/test.nc', encoding={'lat': {'_FillValue': None}, 'lon': {'_FillValue': None}}) as suggested.

  File "/path/to/python3.6/site-packages/xarray/backends/netCDF4_.py", line 354, in prepare_variable
    unlimited_dims=unlimited_dims)
  File "/path/to/python3.6/site-packages/xarray/backends/netCDF4_.py", line 176, in _extract_nc4_variable_encoding
    ' %r' % (backend, invalid))
ValueError: unexpected encoding parameters for 'netCDF4' backend:  ['_FillValue']

Do we just need to append '_FillValue' to the list of

     valid_encodings = set(['zlib', 'complevel', 'fletcher32', 'contiguous',
                           'chunksizes', 'shuffle'])
```

@czr137
Copy link
Contributor Author

czr137 commented Jan 30, 2018

@shoyer, I'll look at the documentation once this is resolved and make a PR.

czr137 added a commit to czr137/xarray that referenced this issue Feb 7, 2018
@czr137 czr137 mentioned this issue Feb 13, 2018
5 tasks
shoyer pushed a commit that referenced this issue Feb 13, 2018
* Add '_FillValue' to set of valid_encodings for netCDF4 backend

* Add additional omit_fill_value tests

Add additional tests to prevent future regression of setting _FillValue
to None when using the encoding kwarg in to_netcdf.

* Fix additional omit_fill_value tests

Remove copy/paste line that shouldn't have been in a test.
Add additional asserts.
Fix indentation.

* Fix scipy failure in additional omit_fill_value tests

* Added bug-fix documentation for #1865
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants