Skip to content

Commit

Permalink
Merge pull request #1130 from mraspaud/feature-datatype-note
Browse files Browse the repository at this point in the history
Add note about datatype in custom reader documentation
  • Loading branch information
djhoese committed Apr 3, 2020
2 parents c35e8c0 + ebbf2f1 commit 2ce07c0
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions doc/source/dev_guide/custom_reader.rst
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,25 @@ a convenience and are not required to read these formats. In many cases using
the :func:`xarray.open_dataset` function in a custom file handler is a much
better idea.

.. note::
Be careful about the data types of the datasets your reader is returning.
It is easy to let the data be coerced into double precision floats (`np.float64`). At the
moment, satellite instruments are rarely measuring in a resolution greater
than what can be encoded in 16 bits. As such, to preserve processing power,
please consider carefully what data type you should scale or calibrate your
data to.

Single precision floats (`np.float32`) is a good compromise, as it has 23
significant bits (mantissa) and can thus represent 16 bit integers exactly,
as well as keeping the memory footprint half of a double precision float.

One commonly used method in readers is :meth:`xarray.DataArray.where` (to
mask invalid data) which can be coercing the data to `np.float64`. To ensure
for example that integer data is coerced to `np.float32` when
:meth:`xarray.DataArray.where` is used, you can do::

my_float_dataarray = my_int_dataarray.where(some_condition, np.float32(np.nan))

One way of implementing a file handler is shown below:

.. code:: python
Expand Down

0 comments on commit 2ce07c0

Please sign in to comment.