New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Representing missing values in string arrays on disk #1647

Open
shoyer opened this Issue Oct 23, 2017 · 1 comment

Comments

Projects
None yet
1 participant
@shoyer
Copy link
Member

shoyer commented Oct 23, 2017

This came up as part of my clean-up of serializing unicode strings in #1648.

There are two ways to represent strings in netCDF files.

  • As character arrays (NC_CHAR), supported by both netCDF3 and netCDF4
  • As variable length unicode strings (NC_STRING), only supported by netCDF4/HDF5.

Currently, by default (if no _FillValue is set) we replace missing values (NaN) with an empty string when writing data to disk.

For character arrays, we could use the normal _FillValue mechanism to set a fill value and decode when data is read back from disk. In fact, this already currently works for dtype=bytes (though it isn't documented):

In [10]: ds = xr.Dataset({'foo': ('x', np.array([b'bar', np.nan], dtype=object), {}, {'_FillValue': b''})})

In [11]: ds
Out[11]:
<xarray.Dataset>
Dimensions:  (x: 2)
Dimensions without coordinates: x
Data variables:
    foo      (x) object b'bar' nan

In [12]: ds.to_netcdf('foobar.nc')

In [13]: xr.open_dataset('foobar.nc').load()
Out[13]:
<xarray.Dataset>
Dimensions:  (x: 2)
Dimensions without coordinates: x
Data variables:
    foo      (x) object b'bar' nan

For variable length strings, it currently isn't possible to set a fill-value. So there's no good way to indicate missing values, though this may change if the future depending on the resolution of the netCDF-python issue.

It would obviously be nice to always automatically round-trip missing values, both for strings and bytes. I see two possible ways to do this:

  1. Require setting an explicit _FillValue when a string contains missing values, by raising an error if this isn't done. We need an explicit choice because there aren't any extra unused characters left over, at least for character arrays. (NetCDF explicitly allows arbitrary bytes to be stored in NC_CHAR, even though this maps to an HDF5 fixed-width string with ASCII encoding.)
    For variable length strings, we could potentially set a non-character unicode symbol like U+FFFF, but again that isn't supported yet.
  2. Treat empty strings as equivalent to a missing value (NaN). This has the advantage of not requiring an explicit choice of _FillValue, so we don't need to wait for any netCDF4 issues to be resolved. However, this does mean that empty strings would not round-trip. Still, given the relative prevalence of missing values vs empty strings in xarray/pandas, it's probably the lesser evil to not preserve empty string.

The default option is to adopt neither of these, and keep the current behavior where missing values are written as empty strings and not decoded at all.

Any opinions? I am leaning towards option (2).

@shoyer

This comment has been minimized.

Copy link
Member

shoyer commented Oct 23, 2017

It occurs to me that yet another option is to avoid using _FillValue:

  1. Instead of using _FillValue, set the missing_value attribute, which is not directly used by netCDF libraries. We could thus choose to let missing_value be a unicode string, and do comparison to find missing values after decoding back into unicode. In contrast, _FillValue is required to be a valid scalar for the encoded netCDF variable, which means a single bytes character when using character encoding. It's pretty awkward for valid choices of _FillValue to change based on which netCDF library / version is being used, so I'd really like to avoid that. (In the current version of #1648, I resolve this issue by encoding unicode fill values into bytes, but this means you need to know that your unicode character encodes into a single byte.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment