Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add note about datatype in custom reader documentation #1130

Merged
merged 2 commits into from
Apr 3, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 19 additions & 0 deletions doc/source/dev_guide/custom_reader.rst
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,25 @@ a convenience and are not required to read these formats. In many cases using
the :func:`xarray.open_dataset` function in a custom file handler is a much
better idea.

.. note::
Be careful about the data types of the datasets your reader is returning.
It is easy to let the data be coerced into double precision floats (`np.float64`). At the
moment, satellite instruments are rarely measuring in a resolution greater
than what can be encoded in 16 bits. As such, to preserve processing power,
please consider carefully what data type you should scale or calibrate your
data to.

Single precision floats (`np.float32`) is a good compromise, as it has 23
significant bits (mantissa) and can thus represent 16 bit integers exactly,
as well as keeping the memory footprint half of a double precision float.

One commonly used method in readers is :meth:`xarray.DataArray.where` (to
mask invalid data) which can be coercing the data to `np.float64`. To ensure
for example that integer data is coerced to `np.float32` when
:meth:`xarray.DataArray.where` is used, you can do::

my_float_dataarray = my_int_dataarray.where(some_condition, np.float32(np.nan))

One way of implementing a file handler is shown below:

.. code:: python
Expand Down