Versions above 1.21.0 will not produce IR Images

[TheMetMan]

Describe the bug
I am decoding MSG Data using SatPy
I upgrade to SatPy 0.46.0 and all the modules in the virtual environment to the latest versions. After that all my images with IR in them would not render, they are just blank.
In particular 'natural_color_with_night_ir' only shows daylight image, not Night. So decoding an image at 0800 hrs of the Eastern Atlantic the cloud in the night part of the image is missing.
After a lot of research I discovered that the problem is with trollimage. Versions above 1.21.0 will not render IR.
To Reproduce

Expected behavior
The image should show the cloud structure for daylight and nighttime.

Actual results
I attach two images to show the difference. The IR part of the image should be rendered, but it is not.

Environment Info:

• OS: Gentoo Linux
• Satpy Version: [e.g. 0.46.0]
  
  

[pnuu]

There's nothing IR-specific in Trollimage. Could you share a minimal script that reproduces the issue you're seeing?

[mraspaud]

Looks like the stretching is not behaving correctly: greenland is blue and the day part also has funky colours.

[TheMetMan]

OK, I am not the only one with this problem, have a look at this link from the Group for Earth Observation MSG

Attached is the Python script to create the above image, my areas yaml file and the list of modules in my Python venv using Python 3.11.7 All the modules are up to date as the latest except trollimage.

msg1_no_comments.py.txt
fg_areas.yaml.txt
modules_list.txt

[djhoese]

My guess is either #145 or #151 and the various PRs that came after 151 to fix some issues.

[pnuu]

Smells like a problem in stretching integer data as this happens only(?) when using a static background image.

[djhoese]

Looking at the second image in the original post I see that the land on the night side still has valid data. The sea/ocean is the only part that seems to be pure black. The night_ir_with_background is a combination of night_ir_alpha and _night_background. When I load just _night_background I don't see any issues (I think). I'll have to track down some SEVIRI data on my computer tonight or tomorrow. Not sure I'll have time this afternoon.

  night_ir_alpha:
    compositor: !!python/name:satpy.composites.GenericCompositor
    standard_name: night_ir_alpha
    prerequisites:
      - name: IR_039
      - name: IR_108
      - name: IR_120
      - name: IR_10

[djhoese]

Ah here we go, why is night_ir_alpha using a dynamic stretch as its enhancement:

  night_ir_alpha:
    standard_name: night_ir_alpha
    operations:
    - name: stretch
      method: !!python/name:satpy.enhancements.stretch
      kwargs: {stretch: linear, cutoffs: [0.02, 0.02]}
    - name: inverse
      method: !!python/name:satpy.enhancements.invert
      args:
      - [true, true, true, true]

If I recall correctly that's exactly what @mraspaud assumed would never be done in #145.

My comment:

Looks good. My only comment on this is what if I want to linear scale the alpha channel? Don't let this question stop this from being merged...but still.

Martin's reply:

I can't really imagine why someone would want to stretch the alpha channel, but yeah, this PR removes that possibility
I'll just merge this, and will fix this if the need to stretch alpha arises.

That need has arisen @mraspaud lol

[mraspaud]

Ok, I'll have a look in the morning

[mraspaud]

@TheMetMan I just created PR #163, which should address part of the problem, the other part being in satpy. I will fix the corresponding satpy recipes once this is merged, but if you want to try it out, in satpy's enhancements/generic.yaml, we should have

  night_ir_alpha:
    standard_name: night_ir_alpha
    operations:
    - name: stretch
      method: !!python/name:satpy.enhancements.stretch
      kwargs: {stretch: linear, cutoffs: [[0.02, 0.02], [0.02, 0.02], [0.02, 0.02], [0.02, 0.02]]}
    - name: inverse
      method: !!python/name:satpy.enhancements.invert
      args:
      - [true, true, true, true]

Using this, I get the expected image.

[TheMetMan]

OK, I have changed the enhancements/generic.yaml file as you suggest, but I get the following error:

Traceback (most recent call last):
  File "/home/francis/FG-Docs/PycharmProjects/Py_MeteosatImageCreator_2/src/msg1_no_comments.py", line 68, in <module>
    compute_writer_results(results)
  File "/home/francis/bin/Python3_11Environments/Py_SatPy_Generic/lib/python3.11/site-packages/satpy/writers/__init__.py", line 594, in compute_writer_results
    da.compute(delayeds)
  File "/home/francis/bin/Python3_11Environments/Py_SatPy_Generic/lib/python3.11/site-packages/dask/base.py", line 663, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/francis/bin/Python3_11Environments/Py_SatPy_Generic/lib/python3.11/site-packages/trollimage/xrimage.py", line 1010, in _compute_quantile
    left, right = data_arr.quantile([cutoffs[0], 1. - cutoffs[1]], dim=['x', 'y'])
                                                 ~~~^~~~~~~~~~~~
TypeError: unsupported operand type(s) for -: 'float' and 'list'

I am using the same script I have uploaded above It happens on the line:

compute_writer_results(results)

[pnuu]

Did you also install the Trollimage version from #163 that adds the handling for the new limits?

[pnuu]

pip install git+https://github.com/mraspaud/trollimage.git@fix-alpha-stretching

should install the version on the PR.

[mraspaud]

@TheMetMan the error you have shows the PR version of trollimage is not installed...

[TheMetMan]

Sorry@mraspaud My error, I have installed trollimage 1.22.2 and then the patch as above and all well now.

[mraspaud]

@TheMetMan thanks a lot for checking this out!