Can't load angle data from msi_safe in version 0.23

[yukaribbba]

Describe the bug
When reading Sentinel-2 MSI L1C dataset, I got error messages saying it can't get any angle data, i.e. solar zenith angle. Atmospheric correction is unable to go through without them.
Everything goes fine in 0.22.

To Reproduce
Download dataset from SciHub

#!/usr/bin/env python
from satpy import Scene, find_files_and_readers
files = find_files_and_readers(base_dir="/mnt/hgfs/Downloads/Sentinel-2/",
reader='msi_safe')
scn = Scene(filenames=files)
composite = 'true_color_marine_clean'
scn.load([composite])
scn.save_dataset(composite, base_dir="/mnt/hgfs/Downloads/Sentinel-2/", writer='geotiff', compress=None)

Expected behavior
Angle data in MSI L1C should be loaded normally.

Actual results
/home/ll/anaconda3/lib/python3.8/site-packages/satpy/resample.py:150: UserWarning: Gradient search resampler not available, upgrade Pyresample.
warnings.warn('Gradient search resampler not available, upgrade Pyresample.')
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
Could not load dataset 'DataID(name='solar_azimuth_angle', resolution=10, modifiers=())': "Could not load DataID(name='solar_azimuth_angle', resolution=10, modifiers=()) from any provided files"
Traceback (most recent call last):
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset
raise KeyError(
KeyError: "Could not load DataID(name='solar_azimuth_angle', resolution=10, modifiers=()) from any provided files"
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
Could not load dataset 'DataID(name='satellite_zenith_angle', resolution=10, modifiers=())': "Could not load DataID(name='satellite_zenith_angle', resolution=10, modifiers=()) from any provided files"
Traceback (most recent call last):
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset
raise KeyError(
KeyError: "Could not load DataID(name='satellite_zenith_angle', resolution=10, modifiers=()) from any provided files"
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
Could not load dataset 'DataID(name='satellite_azimuth_angle', resolution=10, modifiers=())': "Could not load DataID(name='satellite_azimuth_angle', resolution=10, modifiers=()) from any provided files"
Traceback (most recent call last):
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset
raise KeyError(
KeyError: "Could not load DataID(name='satellite_azimuth_angle', resolution=10, modifiers=()) from any provided files"
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
Could not load dataset 'DataID(name='solar_zenith_angle', resolution=10, modifiers=())': "Could not load DataID(name='solar_zenith_angle', resolution=10, modifiers=()) from any provided files"
Traceback (most recent call last):
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset
raise KeyError(
KeyError: "Could not load DataID(name='solar_zenith_angle', resolution=10, modifiers=()) from any provided files"
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
/home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj_string = self.to_proj4()
Traceback (most recent call last):
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/utils.py", line 295, in get_satpos
orb_params = dataset.attrs['orbital_parameters']
KeyError: 'orbital_parameters'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ll/Downloads/s2t.py", line 7, in
scn.load([composite])
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 984, in load
keepables = self._generate_composites()
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 897, in _generate_composites
return self._read_composites(nodes)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 873, in _read_composites
self._generate_composite(item, keepables)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 847, in _generate_composite
composite = compositor(prereq_datasets,
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/composites/init.py", line 570, in call
sata, satz, suna, sunz = self.get_angles(vis)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/composites/init.py", line 552, in get_angles
sat_lon, sat_lat, sat_alt = get_satpos(vis)
File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/utils.py", line 325, in get_satpos
lon = dataset.attrs['satellite_longitude']
KeyError: 'satellite_longitude'

Environment Info:

• OS: Ubuntu 20.04 64bit
• Satpy Version: 0.23
• PyResample Version: 1.16.0
• Readers and writers dependencies (when relevant):
  Readers

abi_l1b: ok
abi_l1b_scmi: ok
abi_l2_nc: ok
acspo: ok
agri_l1: ok
ahi_hrit: ok
ahi_hsd: ok
ami_l1b: ok
amsr2_l1b: ok
amsr2_l2: ok
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
cmsaf-claas2_l2_nc: ok
electrol_hrit: ok
fci_l1c_fdhsi: ok
fci_l2_nc: ok
generic_image: ok
geocat: ok
ghrsst_l3c_sst: ok
glm_l2: ok
goes-imager_hrit: ok
goes-imager_nc: ok
gpm_imerg: ok
grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5: ok
iasi_l2: ok
iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
jami_hrit: ok
li_l2: cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf')
maia: ok
mersi2_l1b: ok
mimicTPW2_comp: ok
modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe: ok
mtsat2-imager_hrit: ok
nucaps: ok
nwcsaf-geo: ok
nwcsaf-msg2013-hdf5: ok
nwcsaf-pps_nc: ok
olci_l1b: ok
olci_l2: ok
omps_edr: ok
safe_sar_l2_ocn: ok
sar-c_safe: ok
scatsat1_l2b: ok
seviri_l1b_hrit: ok
seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native: ok
seviri_l1b_nc: ok
seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b: ok
slstr_l2: ok
smos_l2_wind: ok
tropomi_l2: ok
vaisala_gld360: ok
vii_l1b_nc: ok
vii_l2_nc: ok
viirs_compact: ok
viirs_edr_active_fires: ok
viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b: ok
viirs_sdr: ok
virr_l1b: ok

Writers

cf: ok
geotiff: ok
mitiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi: ok
simple_image: ok

Extras

cartopy: No module named 'cartopy'
geoviews: No module named 'geoviews'

[djhoese]

I think @mraspaud would have the best chance of figuring this out since he was the last one to modify the reader. I think we've seen similar issues in other readers, but this may be the simplest case.

[yukaribbba]

Thanks @djhoese . I just tried to replace the msi_safe.py single file with 0.22 version, but it still doesn’t work. So maybe there are other bugs? Not pretty sure about it.

[djhoese]

Yeah that won't work. There were a lot of changes between versions outside of just the reader file. They can't be swapped like that (at least for these versions).

[simonrp84]

Hello @yukaribbba, this was a bug in how we load the angles. I noticed the same problem and have just created a pull request that fixes this. If you can, please try that version of the code and check that it also works for you.

[yukaribbba]

Thank you everyone, @djhoese @mraspaud @simonrp84 . I've tried the fixed reader and it works.

[simonrp84]

Great!