VIIRS day composite 'snow_age' does not work with Satpy 0.37.1

[lobsiger]

Describe the bug
Dear Developers,

we got the first snow 2022 in Switzerland and 'snow_age' doesn't
work anymore ;-). The attached script has a list of all composites
that used to work with day passes (MX-bands only) in Satpy 0.36.0.
These composites still work in Satpy 0.37.1 except for 'snow_age'.

Attached two 'snow_age' images made with Satpy 0.36.0 and a short
script to reproduce the Satpy 0.37.1 issue and my DEBUG.txt output.

The script including 2 SNPP data segments can be downloaded here:

https://we.tl/t-KG3dU8nAlr 100MBytes *.tgz (link expires in 1 week)

The problem surfaced after EUMETCast users updated PyCoast+Satpy.
It affects may latest distibution of Satpy scripts for EUMETCast.

To Reproduce

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Composite 'snow_age' does not work in SatPy 0.37.1 (it used to work in SatPy 0.36.0)

# We certainly need this
from satpy.scene import Scene
# Why to hell is it not working?
from satpy.utils import debug_on
debug_on()

# Available DAY composites: See files ../satpy/etc/composites/visir.yaml and viirs.yaml
# *************************************************************************************
# ['ash', 'cloudtop_daytime', 'dust', 'false_color', 'fire_temperature', 'fire_temperature_39refl',
#  'fire_temperature_awips', 'fire_temperature_eumetsat', 'fog', 'ir108_3d', 'ir_cloud_day',
#  'natural_color', 'natural_color_sun_lowres', 'natural_with_night_fog', 'night_fog',
#  'ocean_color', 'overview', 'snow_age', 'snow_lowres', 'true_color', 'true_color_crefl',
#  'true_color_lowres', 'true_color_lowres_crefl', 'true_color_lowres_land',
#  'true_color_lowres_marine_tropical', 'true_color_raw']

composite = 'snow_age'

# These two segments should show Italy left and Spain with Gibraltar right (upside down)
files = ['DATA/SVMC_npp_d20221105_t1224383_e1226025_b57121_c20221105123534000154_eum_ops.h5',
         'DATA/SVMC_npp_d20221105_t1226037_e1227279_b57121_c20221105123541000704_eum_ops.h5']

# Swath image size is 3200 X (768) * NumberOfSegments (MX-Bands)
scn = Scene(filenames = files, reader = 'viirs_compact')
scn.load([composite])
scn.save_dataset(composite, 'IMAGES/SNPP-'+composite+'.png')

Expected behavior
Composite 'snow_age' should work as in Satpy 0.36.0

Environment Info:

• OS: Windows and Linux
• Satpy Version: 0.37.1
• PyResample Version: 1.25.1

Suomi-true_color-DEBUG.txt
Suomi-snow_age-DEBUG.txt
Suomi-NPP.py.txt

[djhoese]

This doesn't seem to be a Satpy issue, but rather pillow or dask. Both of these libraries had releases at the end of October. I was unable to reproduce with main branch versions of satpy, pyresample, and trollimage and previous versions of dask and pillow. I will try updating dask and pillow and see what I get.

Edit: Not dask...let's try pillow...

[djhoese]

@lobsiger how did you install Satpy and its dependencies? All conda from conda-forge? Any pip installed packages?

I'm unable to reproduce this on a PopOS (think Ubuntu) with Python 3.10, newest dask and xarray, and the latest pillow I could get. Pillow 9.3.0 is the newest version but not currently available on conda-forge so I am using pillow 9.2.0.

[lobsiger]

@djhoese this is all conda in GNU/Linux IIRC and I have a brand new (yesterday) install under Windows 10 all conda (list attached).
windows.conda.list.txt

[djhoese]

So does it break in Linux, Windows, or both?

[lobsiger]

@djhoese it breaks in both Windows and GNU/Linux. Here is my conda list from Linux.
conda-Linux.list.txt

[lobsiger]

@djhoese and @mraspaud I downgraded pillow and dask one version under GNU/Linux to no avail. I updated again to the latest. Then I downgraded my brand new Windows 10 install with "conda install satpy=0.36.0". Now VIIRS 'snow_age' worked again. I did the same downgrade under GNU/Linux "conda install satpy=0.36.0" and VIIRS 'snow_age' works again. I attach the latest package lists. The conda package lists of these very two PCs before the satpy 0.37.1 downgrade are found in the previous posts.

Windows-downgraded-conda-list.txt
Linux-downgraded-conda-list.txt

[djhoese]

Thanks for trying some things out. I'm still not sure what is going on, but I thought to get the most help from others we should probably post the error message directly in the github issue:

Traceback (most recent call last):
  File "Suomi-NPP.py", line 29, in <module>
    scn.save_dataset(composite, 'IMAGES/SNPP-'+composite+'.png')
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/scene.py", line 1133, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/writers/__init__.py", line 818, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/writers/simple_image.py", line 67, in save_image
    return img.save(filename, compute=compute, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/trollimage/xrimage.py", line 255, in save
    return self.pil_save(filename, fformat, fill_value,
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/trollimage/xrimage.py", line 469, in pil_save
    return delay.compute()
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/base.py", line 315, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/base.py", line 600, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/PIL/Image.py", line 2943, in fromarray
    arr = obj.__array_interface__
AttributeError: 'Array' object has no attribute '__array_interface__'

Looking at what is being called and complained about here, I see fromarray at the bottom. I'm 99% sure that is coming from here:

https://github.com/pytroll/trollimage/blob/c0a30db0895cf6c6c9b63eb0d3c90dfbb890714b/trollimage/xrimage.py#L921

This involves only a few libraries: dask, xarray (res is an xarray DataArray), numpy, pillow, and obviously trollimage which is the package containing this code.

The error message mentions Array which is dask.array.Array. This suggests that dask is either not computing the input dask Array and passing the resulting numpy array or one of the previous steps (np.squeeze) is not wrapping dask arrays properly or one of the previous steps in Satpy (since you said downgrading fixed things) is producing a dask array inside a dask function instead of working with numpy arrays. If I was able to reproduce this it should be fairly simple to nail down what is going on, but at this time I haven't been able to.

@lobsiger Do you see this issue with any other composites or any other readers?

[lobsiger]

@djhoese if I upgrade GNU/Linux again and make a diff of what is installed I get just this

diff Linux-downgraded-conda-list.txt Linux-upgraded-conda-list.txt
satpy 0.36.0 pyhd8ed1ab_0 conda-forge
satpy 0.37.1 pyhd8ed1ab_1 conda-forge

With Satpy 0.37.1 I can do all the DAY composites that I mentioned in my test script except 'snow_age'.
I have not seen similar problems with other readers so far but have not tested all possible composites.

[lobsiger]

@djhoese maybe it helps seeing the error messages of someone else.

https://groups.io/g/MSG-1/message/33545

Is this a reader problem?

[lobsiger]

@djhoese @mraspaud I made an all new latest miniconda install with python 3.10 under GNU/Linux and tested 112 composites of different satellites with satpy 0.37.1. It's still VIIRS 'snow_age' that does not work ...

Linux-latest-miniconda-112-composites.txt
Linux-latest-miniconda-DEBUG-snow_age.txt

[lobsiger]

@djhoese @mraspaud I slightly remembered that we had an issue with MetopC when satpy 0.37.0 was introduced. This lead to satpy 0.37.1. It seems we had the same kind of error message before ... so I think the VIIRS problem is related. See here:

#2173

[djhoese]

That's a good guess. It still doesn't explain why I'm not seeing the issue, but I'll try to find time to play around with it today. Looking into the enhancements like in the MetopC issue is a good start.

[djhoese]

Figured it out! This was fixed in #2190, but it hasn't been released yet. We hope to make a release this week. You could test this in your environment by doing:

pip install git+https://github.com/pytroll/satpy.git

[lobsiger]

This was quick. It's from my bad experience that I shouldn't mix pip and conda installs. I'll rather wait, thanks a lot for your time.

[djhoese]

that I shouldn't mix pip and conda installs

I won't disagree with that. I'll leave this issue open until we make the release.

CC @mraspaud

[lobsiger]

And I take it for granted that this is now not a double fix with #2173 because I cannot play at that Python and Satpy level anyway.

[mraspaud]

Nice catch guys! well done. I'll start preparing a release...

[gerritholl]

Is this still current or can this be closed?