Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

satpy_cf_nc reader fails to read satpy cf writer generated netcdf files where variables start with a number. #1518

Closed
TAlonglong opened this issue Jan 29, 2021 · 3 comments · Fixed by #1525

Comments

@TAlonglong
Copy link
Collaborator

I write netcdf with the satpy cf writer.

Some instruments uses numbers as channel names, like AVHRR channel 1,2,3a,3b,4 and 5. The writer uses these as variable names. According the the cf https://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch02s03.html the variable names should start with a letter.

I try to read back the netcdf files I have created but this fails.
To Reproduce

>>> from satpy import Scene
>>> filenames = ['/home/trygveas/Downloads/noaa18-avhrr-20210129100918-20210129102333.nc', ]
>>> global_scene = Scene(reader='satpy_cf_nc', filenames=filenames)
>>> print(global_scene.available_dataset_names())
['1', '2', '3a', '3b', '4', '5', 'latitude', 'longitude', 'sensor_zenith_angle', 'solar_zenith_angle', 'sun_sensor_azimuth_difference_angle']
>>> print(global_scene.available_dataset_ids())
[DataID(name='1', wavelength=WavelengthRange(min=0.58, central=0.63, max=0.68, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=()), DataID(name='2', wavelength=WavelengthRange(min=0.725, central=0.8625, max=1.0, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=()), DataID(name='3a', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=()), DataID(name='3b', wavelength=WavelengthRange(min=3.55, central=3.74, max=3.93, unit='µm'), resolution=1050, calibration=<calibration.brightness_temperature>, modifiers=()), DataID(name='4', wavelength=WavelengthRange(min=10.3, central=10.8, max=11.3, unit='µm'), resolution=1050, calibration=<calibration.brightness_temperature>, modifiers=()), DataID(name='5', wavelength=WavelengthRange(min=11.5, central=12.0, max=12.5, unit='µm'), resolution=1050, calibration=<calibration.brightness_temperature>, modifiers=()), DataID(name='latitude', modifiers=()), DataID(name='longitude', modifiers=()), DataID(name='sensor_zenith_angle', resolution=1050, modifiers=()), DataID(name='solar_zenith_angle', resolution=1050, modifiers=()), DataID(name='sun_sensor_azimuth_difference_angle', 
![ch1](https://user-images.githubusercontent.com/19924146/106281776-f6eb2780-623f-11eb-8728-c4871a6412e7.jpg)
resolution=1050, modifiers=())]
>>> global_scene.load(['1'])
>>> global_scene.show('1')
<trollimage.xrimage.XRImage object at 0x7f227009f970>
>>> print(global_scene.available_composite_names())
['cloudtop', 'day_microphysics', 'green_snow', 'ir108_3d', 'ir_cloud_day', 'natural_color', 'natural_color_raw', 'natural_enh', 'natural_with_night_fog', 'night_fog', 'night_microphysics', 'overview', 'overview_raw', 'snow']
>>> global_scene.load(['overview'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/scene.py", line 1182, in load
    self.generate_possible_composites(generate, unload)
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/scene.py", line 1239, in generate_possible_composites
    keepables = self._generate_composites_from_loaded_datasets()
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/scene.py", line 1251, in _generate_composites_from_loaded_datasets
    return self._generate_composites_nodes_from_loaded_datasets(nodes)
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/scene.py", line 1257, in _generate_composites_nodes_from_loaded_datasets
    self._generate_composite(node, keepables)
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/scene.py", line 1315, in _generate_composite
    composite = compositor(prereq_datasets,
  File "/home/trygveas/miniconda3/envs/pytroll-weamyl/lib/python3.8/site-packages/satpy/modifiers/geometry.py", line 58, in __call__
    area_name = hash(vis.attrs['area'])
KeyError: 'area'

Test file can be downloaded here https://thredds.met.no/thredds/fileServer/remotesensingsatellite/polar-swath/2021/01/29/noaa18-avhrr-20210129100918-20210129102333.nc

I see that the error is a missing area to the attributes. I dont understand why as the example below works.

Expected behavior
I expected the overview to load.

Environment Info:
linux conda environment, satpy 0.25.1 python 3.8.6

Additional context
Using variable names starting with an letter works better, eg VIIRS data.
same environment:

>>> from satpy import Scene
>>> filenames = ['/home/trygveas/Downloads/noaa20-viirs-mband-20210129094806-20210129100053.nc', ]
>>> global_scene = Scene(reader='satpy_cf_nc', filenames=filenames)
>>> print(global_scene.available_composite_names())
['ash', 'cloudtop_daytime', 'dust', 'false_color', 'fire_temperature_39refl', 'fire_temperature_awips', 'fire_temperature_eumetsat', 'fog', 'ir108_3d', 'ir_cloud_day', 'natural_color', 'natural_color_sun_lowres', 'natural_with_night_fog', 'night_fog', 'ocean_color', 'overview', 'snow_age', 'snow_lowres', 'true_color', 'true_color_lowres', 'true_color_lowres_crefl', 'true_color_lowres_land', 'true_color_lowres_marine_tropical', 'true_color_raw']
>>> global_scene.load(['overview'])
>>> global_scene.show('overview')
<trollimage.xrimage.XRImage object at 0x7f6dc359b0a0>

works fine. Data can be downloaded here: https://thredds.met.no/thredds/fileServer/remotesensingsatellite/polar-swath/2021/01/29/noaa20-viirs-mband-20210129094806-20210129100053.nc

** Suggestion **
Maybe channels starting with or uses only a number as a variable name should be prepended with like channel_1 for channel 1 when writing the netcdf files and the reader could remove this when reading. Or have a special attribute to the variable saying satpy_dataset_name som that satpy can traslate this when reading.

@mraspaud
Copy link
Member

Thanks for digging into this!

I think having the netcdf writer rename the channels to channel_1, channel_2 would be the most intuitive. If we have a name attribute in the metadata of each channel, we could revert to the real name when reading the data.

@TAlonglong do you feel like starting a PR for that?

@djhoese
Copy link
Member

djhoese commented Jan 29, 2021

Should we make a "best practice" that new readers in Satpy shouldn't use channel numbers as channel names? It should always have some prefix? Maybe?

@TAlonglong
Copy link
Collaborator Author

@mraspaud yes, I can at least start one.

@djhoese not sure. In satpy it doesn't matter as far as understand. Only when writing to netcdf/cf you violate the cf recommandation.
But a comment in satpy "best practice" that the user should avoid starting a channel name with a number would not hurt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants