Fix osisaf SST reader

[adybbroe]

Adding support for reading OSISAF GHRSST formattet SST data.
There was some code to read the OSISAF SST but wasn't really finished. Further the SLSTR-L2 reader was actually a GHRSST reader, and both OSISAF and SLSTR-L2 follow the GHRSST format standard. So we can make use of the same code.

• The satpy/etc/readers/ghrsst_l3c_sst.yamlhas been removed and replaced by ghrsst_l2.yaml
• The reader slstr_l2.py has been replaced by ghrsst_l2.py, which is only a slightly modified/enhanced compared to slstr_l2.py
• Renamed the test module test_slstr_l2.py to test_ghrsst_l2.py
• And finally I have added an enhancement in generic.yaml to display the SST using a colormap - I suppose we should remove that and instead suggest how to do this via the documentation or the pytroll-examples?

• Closes #xxxx
• Tests added
• Passes flake8 satpy
• Fully documented

[adybbroe]

I have two yaml files, one for OSISAF (ghrsst_l2.yaml) and one for SLSTR (the old name slstr_l2). Both use the same python code. But the SLSTR one has sensors = slstr obviously. The OSISAF SST is one product but from either VIIRS or AVHRR.
Should we try have one yaml file anyhow, and how do we deal with the sensors attribute then? Or should we keep two yaml files? If yes to the latter I guess we/I should rename them?

[mraspaud]

iirc you can add a sensor property to the file handler, so you would need only one yaml file.

[adybbroe]

Ok, thanks @mraspaud
But wouldn't that mean I would have two entries for the same sea_surface_temperature filed?
Or do you mean something like this:

  sea_surface_temperature_osisaf:
    name: sea_surface_temperature
    coordinates: [longitude_osisaf, latitude_osisaf]
    file_type: GHRSST_OSISAF
    resolution: 2000
    units: kelvin
    standard_name: sea_surface_temperature

  sea_surface_temperature_slstr:
    name: sea_surface_temperature
    sensor: slstr
    coordinates: [longitude_slstr, latitude_slstr]
    file_type: SLSTR
    resolution: 1000
    view: nadir
    units: kelvin
    standard_name: sea_surface_temperature

?

[mraspaud]

no I mean in the filehandler python code

[ghost]

DeepCode's analysis on #1d2e85 found:

• ℹ️ 1 minor issue. 👇

Top issues

Description	Example fixes
standard import "import tarfile" should be placed before "from satpy.readers.file_handlers import BaseFileHandler" Occurrences: ghrsst_l2.py:23 ghrsst_l2.py:24 ghrsst_l2.py:25	🔧 Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot

[adybbroe]

I see what you mean @mraspaud
Not sure though how that is then used with advantage in the yaml file.
Have created one yaml file for both products (OSISAF & SLSTR) ad I can read and project and display both.
But the yaml file seems a bit clumsy still

[adybbroe]

This code now produce the image below:

    scn = Scene(filenames=OSISAF_SST_FILES, reader='ghrsst_l2')
    scn.load(['sea_surface_temperature'])
    local_scn = scn.resample('euron1')
    local_scn.show('sea_surface_temperature')

Using the enhancement here:

  osisaf_sst_smhi_colormap:
    standard_name: sea_surface_subskin_temperature
    operations:
    - name: osisaf_sst
      method: !!python/name:satpy.enhancements.colorize
      kwargs:
          palettes:
            - colors: [
              [255, 0, 255],
              [195, 0, 129],
              [129, 0, 47],
              [195, 0, 0],
              [255, 0, 0],
              [236, 43, 0],
              [217, 86, 0],
              [200, 128, 0],
              [211, 154, 13],
              [222, 180, 26],
              [233, 206, 39],
              [244, 232, 52],
              [255.99609375, 255.99609375, 63.22265625],
              [203.125, 255.99609375, 52.734375],
              [136.71875, 255.99609375, 27.34375],
              [0, 255.99609375, 0],
              [0, 207.47265625, 0],
              [0, 158.94921875, 0],
              [0, 110.42578125, 0],
              [0, 82.8203125, 63.99609375],
              [0, 55.21484375, 127.9921875],
              [0, 27.609375, 191.98828125],
              [0, 0, 255.99609375],
              [100.390625, 100.390625, 255.99609375],
              [150.5859375, 150.5859375, 255.99609375]]
              min_value: 296.55
              max_value: 273.55

It is supposed to look as similar as possible to the imagery we currently produce at SMHI with mpop. The corresponding image and its diff is added below:

[adybbroe]

I have added a PR (#2075) documenting how the use of a colorize enhancement as above can be done. So I am ready to remove the part in the enhancement.yaml file. I have, however, also asked the OSISAF if they could confirm the coloring above or provide what they regard as the standard. In which case I think it would be fair to include it among the default enhancements. Agree?

[adybbroe]

Got an answer from our colleagues in the OSISAF. There is yet no standard/official OSISAF color scale to refer to, so I suggest we remove it from the general.yaml file in this PR.

[mraspaud]

Thanks for this PR, looks promising. Some inline comments.

[codecov]

Codecov Report

Merging #1410 (e3d7dda) into main (d4c4332) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1410      +/-   ##
==========================================
+ Coverage   93.86%   93.89%   +0.02%     
==========================================
  Files         283      283              
  Lines       42379    42583     +204     
==========================================
+ Hits        39781    39985     +204     
  Misses       2598     2598              

Flag	Coverage Δ
behaviourtests	4.70% <0.00%> (-0.03%)	⬇️
unittests	94.45% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
satpy/composites/__init__.py	89.94% <ø> (ø)
satpy/readers/ghrsst_l2.py	100.00% <100.00%> (ø)
satpy/tests/reader_tests/test_ghrsst_l2.py	100.00% <100.00%> (ø)
satpy/readers/ghrsst_l3c_sst.py	0.00% <0.00%> (-7.05%)	⬇️
satpy/modifiers/_crefl.py	92.40% <0.00%> (-0.62%)	⬇️
satpy/tests/test_crefl_utils.py	100.00% <0.00%> (ø)
satpy/readers/seviri_l1b_hrit.py	90.14% <0.00%> (ø)
satpy/tests/reader_tests/test_utils.py	100.00% <0.00%> (ø)
satpy/tests/reader_tests/test_fci_l1c_nc.py	100.00% <0.00%> (ø)
satpy/tests/reader_tests/test_viirs_l1b.py	97.26% <0.00%> (+0.06%)	⬆️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4c4332...e3d7dda. Read the comment docs.

[coveralls]

Coverage increased (+0.03%) to 94.397% when pulling e3d7dda on adybbroe:fix-osisaf-reader into d4c4332 on pytroll:main.

[adybbroe]

I tested the reader on the SLSTR files of this type: S3A_SL_2_WST____20220406T095733_20220406T100033_20220406T120437_0179_084_022_2340_ MAR_O_NR_003.SEN3.tar

and on the OSISAF netCDF files of this type: S-OSI_-FRA_-MTOP-NARSST_FIELD-202204061000Z.nc

Here are the results of both at the approximate same time morning April 6th, 2022 over Scandinavia (SLSTR to the right):

[sfinkens]

Nice work! Just a couple questions about readability.

[sfinkens]

You could move the if condition to a separate method, _is_sst_file for instance

[adybbroe]

Good suggestion, I will fix

[sfinkens]

Instead of mocking, would it be possible to write such a tarfile during test setup?

[adybbroe]

Yes, I think this is a good idea. I will come back with a proposal

[sfinkens]

LGTM, thanks for refactoring the tests! Just two tiny comments

[adybbroe]

@mraspaud Are my updates satisfactory do you think?

[mraspaud]

yes, it look good now, but the tests are failing on windows, because of the classical permission error. However, as we are moving to pytest more now, I would suggest to transition the tests to pytest here too, and use the tmp_path fixture to handle temporary data and let pytest handle the removal of the temporary files/directories to come around the problem.

[adybbroe]

Okay, so now the tests pass! There is a 0.02% decrease in test coverage. But the Coveralls page is not very friendly - don't see where this decrease sits?
I am off tomorrow, but hopefully it is good enough?

[mraspaud]

LGTM, much nicer with pytest!