Preservation of input data dtype in processing FCI data

[pnuu]

Recently we've been creating a lot of pull requests to make Satpy preserve the data type of the FCI input data as long as possible. The main thing is not to convert float32 data to float64 without a good reason to make processing to use less memory and be faster in general when the arrays are not converted back and forth.

This issue collects all of the issues/pull requests together.

Readers

Satpy

• fci_l1c_nc: Keep FCI data as 32-bit floats #2637
  • raw data are stored as 16 bit integers, 32-bit floats to be used for calibrated data

Compositors and related packages

Satpy

• DayNightCompositor: Keep original dtype in DayNightCompositor #2640
  • was casting everything to float64
  • the default stretch applied in the tests also changed dtype, see below

Enhancements and related packages

Trollimage

• linear_stretch/crude_stretch: Keep the original dtype of the data when stretching trollimage#150
  • this is applied to all composites by default, so forced all composites saved without using enhance=False to be converted to float64 before saving (and again converted to uint8)
  • test added also for invert() enhancement
• gamma: Preserve dtypes in XRImage "enhancements" trollimage#151
  • tests added for also all the other enhancements not covered by trollimage/150

Modifiers and related packages

Satpy

• blackbody radiance to brightness temperature conversion: modifier NIREmissivePartFromReflectance triggers early dask computation #2613 -> Allow dataarrays and preserve dtype in rad2temp pyspectral#203
• NIR reflectance: Set dtype for get_lonlats() in NIR reflectance calculation #2642
  • makes sure Sun zenith angles calculated for the reflectance retrieval match the dtype of the NIR channel

Pyspectral

• blackbody radiance to brightness temperature conversion: blackbody functions not dask friendly, trigger early dask computation pyspectral#201 -> Allow dataarrays and preserve dtype in rad2temp pyspectral#203
  • also allows xr.DataArrays as inputs
• NIR reflectance and emissivity: Near infrared reflectance calculation does not preserve input dtype pyspectral#205 -> Return 3.x um reflectance with the same dtype as the NIR data pyspectral#206
  • also further updates to blackbody calculations used by the NIR classes

[pnuu]

I also checked that saving in trollimage.xrimage.XRImage doesn't do any up-casting internally, but this is not tested as far as I can tell.

[pnuu]

Trollimage and Pyspectral have been released with the above updates, Satpy release remaining.

[mraspaud]

@pnuu could you make a short summary here of the performance benefits of this work?

[pnuu]

I'll run a benchmark with the releases before these changes were made and with the latest versions (main for Satpy) and post in a bit.

[pnuu]

Benchmarks.

For both cases:

• Dask workers: 4
• OMP threads: 1
• GDAL threads: 4
• Dask chunk size: 2097152
• FCI L1c data, both FDHSI and HRFI file supplied
• 3 RGB composites using GenericCompositor
• 1 composite using GenericCompositor saved as float32
• 7 composites using DayNightCompositor, only day or night side shown, the other side is transparent
• load the composites with compute=False
• resampling gradient_search
• saved as COG images

Using previous released versions:

• Trollimage: 1.21.0
• Pyspectral: 0.12.5
• Satpy: 0.44.0

With the changes listed in this issue:

• Trollimage: 1.22.2
• Pyspectral: 1.13.0
• Satpy main, commit a1c3b13

[pnuu]

I noticed that there were two PRs not listed here that affected the performance:

• Reduce Dask computations in DayNightCompositor #2617
• Remove duplicate entries of required netcdf variables in FCI reader #2634

I'll run the test with these fixes incrementally keeping the other packages at their previous release versions.

[pnuu]

Without #2617 and #2634 (commit f189402): similar to v0.44.0

With #2617 (commit 86c075a):

With #2634 (commit e43f3fa):

[pnuu]

Meh, tracking the merge order is too tedious to see what happened when. In any case, the removal of duplicates from the FCI reader config shouldn't have that much of a effect compared to the previous case.

[pnuu]

Still missed one: #2623 . I bet this was merged between #2617 and #2634 and made the big jump in performance.

[pnuu]

Yep, that explains the jump.

Before #2623 (commit 14f59c4)

With #2623 (commit d59e467)

[mraspaud]

Thanks for driving this @pnuu !

[ameraner]

Summarising, if I see this correctly, with satpy>=v0.45.0, pyspectral>=v0.13.0 and trollimage>=1.22.2 one should get all improvements

[pnuu]

Summarising, if I see this correctly, with satpy>=v0.45.0, pyspectral>=v0.13.0 and trollimage>=1.22.2 one should get all improvements

Yes, that's correct.