'ModuleNotFoundError' when processing Sentinel-2 MSI data

[yukaribbba]

Describe the bug
Error occured, no image output.

To Reproduce

#!/usr/bin/env python
from satpy import Scene, find_files_and_readers
files = find_files_and_readers(base_dir="/mnt/hgfs/Downloads/Sat/",
                               reader='msi_safe')
scn = Scene(filenames=files)
composite = 'true_color_marine_clean'
scn.load([composite])
scn.save_dataset(composite, filename='{name}_{start_time:%Y%m%d_%H%M%S}.tif', writer='geotiff', base_dir="/mnt/hgfs/Downloads/Sat/", num_threads=8, compress=None, fill_value=0)

Expected behavior
Should go through smoothly.

Actual results
/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/msi_safe.py:199: RuntimeWarning: Mean of empty slice
return np.nanmean(np.dstack(arrays), -1)
Traceback (most recent call last):
File "/home/ll/Downloads/s2t.py", line 7, in
scn.load([composite])
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/scene.py", line 1154, in load
self._read_datasets_from_storage(**kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/scene.py", line 1174, in _read_datasets_from_storage
return self._read_dataset_nodes_from_storage(nodes, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/scene.py", line 1180, in _read_dataset_nodes_from_storage
loaded_datasets = self._load_datasets_by_readers(reader_datasets, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/scene.py", line 1205, in _load_datasets_by_readers
new_datasets = reader_instance.load(ds_ids, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 945, in load
ds = self._load_dataset_with_area(dsid, coords, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 689, in _load_dataset
projectable = fh.get_dataset(dsid, ds_info)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/satpy/readers/msi_safe.py", line 210, in get_dataset
darr = darr.bfill('x')
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/dataarray.py", line 2570, in bfill
return bfill(self, dim, limit=limit)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/missing.py", line 430, in bfill
return apply_ufunc(
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/computation.py", line 1171, in apply_ufunc
return apply_dataarray_vfunc(
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/computation.py", line 288, in apply_dataarray_vfunc
result_var = func(*data_vars)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/computation.py", line 739, in apply_variable_ufunc
result_data = func(*input_data)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/missing.py", line 400, in _bfill
arr = push(arr, axis=axis, n=n)
File "/home/ll/anaconda3/envs/env/lib/python3.9/site-packages/xarray/core/duck_array_ops.py", line 660, in push
from bottleneck import push
ModuleNotFoundError: No module named 'bottleneck'

Environment Info:

• OS: Ubuntu 20.04 64bit
• Satpy Version: 0.29

[djhoese]

How did you create your environment and install the dependencies? The bottleneck library should be a dependency of xarray and/or dask. It is possible that changes to these libraries have lost the bottleneck dependency.

[djhoese]

Looking at my own usual satpy environment, I don't see bottleneck installed. This means there must be something that the msi_safe reader is doing with xarray that is going down a weird path, or you have an old version of xarray, or an old version of dask maybe. What versions of xarray, dask, and numpy do you have installed?

@mraspaud Are you able to use this reader? Do you have bottleneck installed?

[mraspaud]

I can reproduce this. In a fresh environment, bottleneck doesn't get installed, but the bfill call in the msi_safe reader crashed if it's not there

[mraspaud]

The xarray documentation names this, so this is a bug in satpy. http://xarray.pydata.org/en/stable/generated/xarray.DataArray.bfill.html

I'm on it

[mraspaud]

@yukaribbba thanks for reporting the issue, this is solved in #1723 I think.

[yukaribbba]

@djhoese
Yes just as @mraspaud said, I'm working on a new machine. I set the environment by conda install -c conda-forge satpy --all

[yukaribbba]

@mraspaud Great, after bottleneck installed everything goes fine. Thank you!