Loading both abi and nwcsaf-geo confuses satpy into sometimes trying the wrong composite #1112

gerritholl · 2020-03-12T12:34:16Z

Describe the bug

When I load both ABI data and NWCSAF-GEO data, satpy loads composites files from abi.yaml, visir.yaml, and seviri.yaml (with #1111, also ahi.yaml). When I subsequently try to load a composite such as overview, depending on which composite file was loaded first, it either succeeds (if abi.yaml was loaded first) or fails (if seviri.yaml was loaded first).

To Reproduce

# Your code here
import satpy
from satpy.utils import debug_on
debug_on()
sc = satpy.Scene(
    {"abi_l1b": ['import/Sat_data/OR_ABI-L1b-RadF-M3C02_G16_s20170732006100_e20170732016467_c20170732016504.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C03_G16_s20170732006100_e20170732016467_c20170732016515.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C04_G16_s20170732006100_e20170732016467_c20170732016500.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C05_G16_s20170732006100_e20170732016467_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C06_G16_s20170732006100_e20170732016473_c20170732016511.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C08_G16_s20170732006100_e20170732016467_c20170732016512.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C09_G16_s20170732006100_e20170732016473_c20170732016528.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C10_G16_s20170732006100_e20170732016478_c20170732016529.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C11_G16_s20170732006100_e20170732016467_c20170732016525.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C13_G16_s20170732006100_e20170732016479_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C14_G16_s20170732006100_e20170732016467_c20170732016530.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C15_G16_s20170732006100_e20170732016473_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C16_G16_s20170732006100_e20170732016479_c20170732016530.nc'],
     "nwcsaf-geo": ['export/CMIC/S_NWC_CMIC_GOES16_NEW-ENGLAND-NR_20170314T200610Z.nc']})
sc.load(["overview"])

Expected behavior

I expect that this code consistently executes without error messages.

Actual results
Text output of actual results or error messages including full tracebacks if applicable.

Sometimes it fails, sometimes it succeeds. This appear to depend on the order in which the composites are loaded, which appears non-deterministic.

Example of when it succeeds (abi.yaml loaded before seviri.yaml):

[DEBUG: 2020-03-12 10:40:43 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc'
[DEBUG: 2020-03-12 10:40:43 : satpy.readers] Reading ['/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2020-03-12 10:40:43 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['import/Sat_data/OR_ABI-L1b-RadF-M3C02_G16_s20170732006100_e20170732016467_c20170732016504.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C03_G16_s20170732006100_e20170732016467_c20170732016515.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C04_G16_s20170732006100_e20170732016467_c20170732016500.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C05_G16_s20170732006100_e20170732016467_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C06_G16_s20170732006100_e20170732016473_c20170732016511.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C08_G16_s20170732006100_e20170732016467_c20170732016512.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C09_G16_s20170732006100_e20170732016473_c20170732016528.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C10_G16_s20170732006100_e20170732016478_c20170732016529.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C11_G16_s20170732006100_e20170732016467_c20170732016525.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C13_G16_s20170732006100_e20170732016479_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C14_G16_s20170732006100_e20170732016467_c20170732016530.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C15_G16_s20170732006100_e20170732016473_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C16_G16_s20170732006100_e20170732016479_c20170732016530.nc']
[DEBUG: 2020-03-12 10:40:44 : satpy.readers] Reading ['/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/nwcsaf-geo.yaml']
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.yaml_reader] Assigning to nwcsaf-geo: ['export/CMIC/S_NWC_CMIC_GOES16_NEW-ENGLAND-NR_20170314T200610Z.nc']
[DEBUG: 2020-03-12 10:40:44 : satpy.composites] Looking for composites config file abi.yaml
[DEBUG: 2020-03-12 10:40:44 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2020-03-12 10:40:44 : satpy.composites] Looking for composites config file seviri.yaml
[DEBUG: 2020-03-12 10:40:44 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[DEBUG: 2020-03-12 10:40:44 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:539: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Reading in get_dataset C14.
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Calibrating to brightness temperatures
/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:539: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Reading in get_dataset C02.
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Calibrating to reflectances
/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:539: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Reading in get_dataset C03.
[DEBUG: 2020-03-12 10:40:44 : satpy.readers.abi_l1b] Calibrating to reflectances
[DEBUG: 2020-03-12 10:40:44 : satpy.composites] Applying sun zen correction
[DEBUG: 2020-03-12 10:40:44 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Sun-zenith correction applied. Computation time:   0.2 (sec)
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Applying sun zen correction
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2020-03-12 10:40:45 : satpy.composites] Sun-zenith correction applied. Computation time:   0.2 (sec)
[DEBUG: 2020-03-12 10:40:45 : satpy.scene] Delaying generation of DatasetID(name='overview', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of incompatible areas
[WARNING: 2020-03-12 10:40:45 : satpy.scene] The following datasets were not created and may require resampling to be generated: DatasetID(name='overview', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)
[DEBUG: 2020-03-12 10:40:45 : satpy.scene] Unloading dataset: DatasetID(name='C02', wavelength=(0.59, 0.64, 0.69), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=())
[DEBUG: 2020-03-12 10:40:45 : satpy.scene] Unloading dataset: DatasetID(name='C03', wavelength=(0.8455, 0.865, 0.8845), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())

Example of when it fails (seviri.yaml loaded before abi.yaml):

[DEBUG: 2020-03-12 10:40:35 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc'
[DEBUG: 2020-03-12 10:40:35 : satpy.readers] Reading ['/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2020-03-12 10:40:35 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['import/Sat_data/OR_ABI-L1b-RadF-M3C02_G16_s20170732006100_e20170732016467_c20170732016504.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C03_G16_s20170732006100_e20170732016467_c20170732016515.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C04_G16_s20170732006100_e20170732016467_c20170732016500.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C05_G16_s20170732006100_e20170732016467_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C06_G16_s20170732006100_e20170732016473_c20170732016511.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C08_G16_s20170732006100_e20170732016467_c20170732016512.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C09_G16_s20170732006100_e20170732016473_c20170732016528.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C10_G16_s20170732006100_e20170732016478_c20170732016529.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C11_G16_s20170732006100_e20170732016467_c20170732016525.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C13_G16_s20170732006100_e20170732016479_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C14_G16_s20170732006100_e20170732016467_c20170732016530.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C15_G16_s20170732006100_e20170732016473_c20170732016531.nc', 'import/Sat_data/OR_ABI-L1b-RadF-M3C16_G16_s20170732006100_e20170732016479_c20170732016530.nc']
[DEBUG: 2020-03-12 10:40:36 : satpy.readers] Reading ['/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/nwcsaf-geo.yaml']
[DEBUG: 2020-03-12 10:40:36 : satpy.readers.yaml_reader] Assigning to nwcsaf-geo: ['export/CMIC/S_NWC_CMIC_GOES16_NEW-ENGLAND-NR_20170314T200610Z.nc']
[DEBUG: 2020-03-12 10:40:36 : satpy.composites] Looking for composites config file seviri.yaml
[DEBUG: 2020-03-12 10:40:36 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2020-03-12 10:40:36 : satpy.composites] Looking for composites config file abi.yaml
Traceback (most recent call last):
  File "/home/gholl/mwe00.py", line 7, in <module>
    sc.load(["overview"])
  File "/opt/pytroll/pytroll_inst/miniconda3-exp/envs/py38/lib/python3.8/site-packages/satpy/scene.py", line 968, in load
    raise KeyError("Unknown datasets: {}".format(unknown_str))
KeyError: "Unknown datasets: DatasetID(name='VIS008', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=()), IR_108, DatasetID(name='VIS006', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())"

Environment Info:

OS: CentOS Linux release 7.6.1810 (Core)
Satpy Version: v0.20.0-47-ge91b6c7b
PyResample Version: 1.14.0
Readers and writers dependencies (when relevant):

Readers ======= abi_l1b: ok abi_l1b_scmi: ok abi_l2_nc: ok acspo: ok agri_l1: ok ahi_hrit: ok ahi_hsd: ok ami_l1b: ok amsr2_l1b: ok avhrr_l1b_aapp: ok avhrr_l1b_eps: ok avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac') avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac') caliop_l2_cloud: ok clavrx: ok electrol_hrit: ok fci_l1c_fdhsi: ok generic_image: ok geocat: ok ghrsst_l3c_sst: ok glm_l2: ok goes-imager_hrit: ok goes-imager_nc: ok grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib') hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib') iasi_l2: ok iasi_l2_so2_bufr: ok jami_hrit: ok li_l2: ok maia: ok mersi2_l1b: ok mimicTPW2_comp: ok modis_l1b: ok modis_l2: ok msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur') mtsat2-imager_hrit: ok nucaps: ok nwcsaf-geo: ok nwcsaf-msg2013-hdf5: ok nwcsaf-pps_nc: ok olci_l1b: ok olci_l2: ok omps_edr: ok safe_sar_l2_ocn: ok sar-c_safe: ok scatsat1_l2b: ok seviri_l1b_hrit: ok seviri_l1b_icare: ok seviri_l1b_native: ok seviri_l1b_nc: ok seviri_l2_bufr: ok slstr_l1b: ok slstr_l2: ok tropomi_l2: ok vaisala_gld360: ok viirs_compact: ok viirs_edr_active_fires: ok viirs_edr_flood: ok viirs_l1b: ok viirs_sdr: ok virr_l1b: ok

Writers

cf: ok
geotiff: ok
mitiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi: ok
simple_image: ok

Extras

cartopy: ok
geoviews: ok

Additional context

PR #1111 explicitly lists abi and ahi as sensors for which the NWCSAF-GEO can work, this does not solve the problem.

The text was updated successfully, but these errors were encountered:

gerritholl · 2020-03-12T12:38:06Z

@mraspaud said on slack that @adybbroe has seen this bug, but I could not find it in the issues list. My apologies if this is a duplicate.

djhoese · 2020-03-12T12:49:26Z

While I acknowledge this is a real problem, I'm curious why you are loading these together in the same Scene? One quick workaround for the ABI versus NWCSAF-GEO composites is to load them separately. That doesn't help in the case of a reader that lists multiple sensors though.

gerritholl · 2020-03-12T12:51:53Z

I'm loading them together because the fogpy fls_day composite for ABI needs both:

  fls_day:
    compositor: !!python/name:fogpy.composites.FogCompositorDay
    prerequisites:
      - C02
      - C03
      - C05
      - C07
      - C11
      - C14
      - C15
      - cmic_cot
      - cmic_lwp
      - cmic_reff
    standard_name: fls_day
    path_dem: /media/nas/x21308/DEM/dem_eu_1km.tif

https://github.com/gerritholl/fogpy/blob/4af46cee743c72ff629a8754b669dcb4104cce5d/etc/composites/abi.yaml#L12-L26

djhoese · 2020-03-12T12:55:02Z

So the not super-easy workaround:

scn_nwcsaf = Scene(...)
scn_nwcsaf.load(['cmic_cot', 'cmic_lwp', 'cmic_reff'])
scn_abi['cmic_cot'] = scn_nwcsaf['cmic_cot']
scn_abi['cmic_lwp'] = scn_nwcsaf['cmic_lwp']
scn_abi['cmic_reff'] = scn_nwcsaf['cmic_reff']
scn_abi.load(['fls_day'])

pnuu · 2020-03-12T12:57:05Z

I'm curious why you are loading these together in the same Scene?

This is also needed for the MaskingCompositor, where the data really need to be loaded at the same time.

adybbroe · 2020-03-12T13:02:05Z

I'm curious why you are loading these together in the same Scene?

This is also needed for the MaskingCompositor, where the data really need to be loaded at the same time.

Yes agree.

adybbroe · 2020-03-12T13:03:27Z

@mraspaud said on slack that @adybbroe has seen this bug, but I could not find it in the issues list. My apologies if this is a duplicate.

Correct. I didn't yet come to make the issue @gerritholl. I found the bug/issue while preparing my presentation for yesterday, so was too busy. Thanks for posting it. Sorry for discovering it again!

djhoese · 2020-03-12T13:04:04Z

We have talked in the past about possibly defining multi-sensor composites in these special cases like viirs-abi.yaml or something. Not sure that is the best solution but it is an idea.

As mentioned on slack, the correct fix for this particular case is to fix the nwcsaf-geo reader's sensor_names property to only return the sensor for the files provided.

adybbroe · 2020-03-12T13:20:14Z

satpy/satpy/readers/nwcsaf_nc.py

Lines 80 to 92 in e91b6c7

    
           try: 
        
               # MSG: 
        
               sat_id = self.nc.attrs['satellite_identifier'] 
        
               try: 
        
                   self.platform_name = PLATFORM_NAMES[sat_id] 
        
               except KeyError: 
        
                   self.platform_name = PLATFORM_NAMES[sat_id.astype(str)] 
        
           except KeyError: 
        
               # PPS: 
        
               self.platform_name = self.nc.attrs['platform'] 
        
               self.pps = True 
        
           self.sensor = SENSOR.get(self.platform_name, 'seviri')

@djhoese As far as I can see this is what is tried here?

djhoese · 2020-03-12T13:30:46Z

That won't work for this particular problem. The .sensor property you have here doesn't mean anything to the base reader in yaml_reader which is what's provide the Scene with the information used to load composites.

There needs to be a sensor_names property in the file handlers like this.
The base reader then uses that here.
The Scene then uses that property from the reader here.

gerritholl · 2020-03-12T13:37:32Z

As far as fogpy is concerned, it's not actually a multi-sensor composite, but it's still a multi-reader composite. But multiple sensors on the same platform should also be possible now that not only LEO but also GEO satellites are starting to have a more diverse set of instruments.

adybbroe mentioned this issue Mar 15, 2020

Add sensor-name property to NWCSAF readers #1113

Merged

6 tasks

mraspaud closed this as completed in #1113 Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loading both abi and nwcsaf-geo confuses satpy into sometimes trying the wrong composite #1112

Loading both abi and nwcsaf-geo confuses satpy into sometimes trying the wrong composite #1112

gerritholl commented Mar 12, 2020

Writers

Extras

gerritholl commented Mar 12, 2020

djhoese commented Mar 12, 2020

gerritholl commented Mar 12, 2020 •

edited

djhoese commented Mar 12, 2020

pnuu commented Mar 12, 2020

adybbroe commented Mar 12, 2020

adybbroe commented Mar 12, 2020 •

edited

djhoese commented Mar 12, 2020

adybbroe commented Mar 12, 2020

djhoese commented Mar 12, 2020

gerritholl commented Mar 12, 2020

Loading both abi and nwcsaf-geo confuses satpy into sometimes trying the wrong composite #1112

Loading both abi and nwcsaf-geo confuses satpy into sometimes trying the wrong composite #1112

Comments

gerritholl commented Mar 12, 2020

Writers

Extras

gerritholl commented Mar 12, 2020

djhoese commented Mar 12, 2020

gerritholl commented Mar 12, 2020 • edited

djhoese commented Mar 12, 2020

pnuu commented Mar 12, 2020

adybbroe commented Mar 12, 2020

adybbroe commented Mar 12, 2020 • edited

djhoese commented Mar 12, 2020

adybbroe commented Mar 12, 2020

djhoese commented Mar 12, 2020

gerritholl commented Mar 12, 2020

gerritholl commented Mar 12, 2020 •

edited

adybbroe commented Mar 12, 2020 •

edited