Fix padding of segmented geostationary images #1021

sfinkens · 2019-12-17T14:49:33Z

I noticed some problems with the new padding of geostationary images:

ahi_hsd does not recognize segment from the filename and expected_segments from YAML
mtsat2-imager_hrit doesn't work anymore with non-segmented data (no segment key in the filename info)

Reading multiple scans at once doesn't work anymore:

In [1]: import satpy 
...: import glob 
...: from satpy.utils import debug_on                                                                                                                                                                                                     

In [2]: debug_on()                                                                                                                                                                                                                           

In [3]: scene = satpy.Scene(filenames=glob.glob('test_data/mtsat2_hrit/*'), reader='mtsat2-imager_hrit')                                                                                                                                     
[DEBUG: 2019-12-17 14:42:36 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/etc'
[DEBUG: 2019-12-17 14:42:36 : satpy.readers] Reading ['/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/etc/readers/mtsat2-imager_hrit.yaml']
[DEBUG: 2019-12-17 14:42:36 : satpy.readers.yaml_reader] Assigning to mtsat2-imager_hrit: ['test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0601_DK03IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1901_DK02IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0101_DK02IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1832_DK01IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1232_DK01IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0032_DK01IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0701_DK02IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0001_DK03IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1801_DK03IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1201_DK03IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_1301_DK02IR1', 'test_data/mtsat2_hrit/HRIT_MTSAT2_20140524_0632_DK01IR1']
[DEBUG: 2019-12-17 14:42:36 : satpy.composites] Looking for composites config file mtsat2_imager.yaml
[DEBUG: 2019-12-17 14:42:36 : satpy.composites] No composite config found called mtsat2_imager.yaml

In [4]: scene.load(['IR1'])                                                                                                                                                                                                                  
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.005258
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.002388
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.001838
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.001775
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.003688
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.003702
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.002927
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.003964
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.003751
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.002497
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.003398
[DEBUG: 2019-12-17 14:42:40 : hrit_jma] Calibration time 0:00:00.002341
[DEBUG: 2019-12-17 14:42:40 : satpy.resample] Could not add 'crs' coordinate with pyproj<2.0
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-d0ce341080f8> in <module>
----> 1 scene.load(['IR1'])

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs)
    971             raise KeyError("Unknown datasets: {}".format(unknown_str))
    972 
--> 973         self.read(**kwargs)
    974         if generate:
    975             keepables = self.generate_composites()

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/scene.py in read(self, nodes, **kwargs)
    880             required_nodes = self.wishlist - set(self.datasets.keys())
    881             nodes = self.dep_tree.leaves(nodes=required_nodes)
--> 882         return self._read_datasets(nodes, **kwargs)
    883 
    884     def generate_composites(self, nodes=None):

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/scene.py in _read_datasets(self, dataset_nodes, **kwargs)
    721         for reader_name, ds_ids in reader_datasets.items():
    722             reader_instance = self.readers[reader_name]
--> 723             new_datasets = reader_instance.load(ds_ids, **kwargs)
    724             loaded_datasets.update(new_datasets)
    725         self.datasets.update(loaded_datasets)

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/readers/yaml_reader.py in load(self, dataset_keys, previous_datasets, **kwargs)
    838             coords = [all_datasets.get(cid, None)
    839                       for cid in coordinates.get(dsid, [])]
--> 840             ds = self._load_dataset_with_area(dsid, coords, **kwargs)
    841             if ds is not None:
    842                 all_datasets[dsid] = ds

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/readers/yaml_reader.py in _load_dataset_with_area(self, dsid, coords, **kwargs)
    755         if area is not None:
    756             ds.attrs['area'] = area
--> 757             ds = add_crs_xy_coords(ds, area)
    758         return ds
    759 

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/resample.py in add_crs_xy_coords(data_arr, area)
    300     else:
    301         # Gridded data (AreaDefinition/StackedAreaDefinition)
--> 302         data_arr = add_xy_coords(data_arr, area, crs=crs)
    303     return data_arr
    304 

/cmsaf/nfshome/sfinkens/software/devel/satpy/satpy/resample.py in add_xy_coords(data_arr, area, crs)
    245     y = xr.DataArray(y, dims=('y',), attrs=y_attrs)
    246     x = xr.DataArray(x, dims=('x',), attrs=x_attrs)
--> 247     return data_arr.assign_coords(y=y, x=x)
    248 
    249 

/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/satpy/lib/python3.7/site-packages/xarray/core/common.py in assign_coords(self, coords, **coords_kwargs)
    457         data = self.copy(deep=False)
    458         results = self._calc_assign_results(coords_kwargs)
--> 459         data.coords.update(results)
    460         return data
    461 

/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/satpy/lib/python3.7/site-packages/xarray/core/coordinates.py in update(self, other)
    116             [self.variables, other_vars], priority_arg=1, indexes=self.indexes
    117         )
--> 118         self._update_coords(coords, indexes)
    119 
    120     def _merge_raw(self, other):

/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/satpy/lib/python3.7/site-packages/xarray/core/coordinates.py in _update_coords(self, coords, indexes)
    289         coords_plus_data = coords.copy()
    290         coords_plus_data[_THIS_ARRAY] = self._data.variable
--> 291         dims = calculate_dimensions(coords_plus_data)
    292         if not set(dims) <= set(self.dims):
    293             raise ValueError(

/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/satpy/lib/python3.7/site-packages/xarray/core/dataset.py in calculate_dimensions(variables)
    187                     "conflicting sizes for dimension %r: "
    188                     "length %s on %r and length %s on %r"
--> 189                     % (dim, size, k, dims[dim], last_used[dim])
    190                 )
    191     return dims

ValueError: conflicting sizes for dimension 'y': length 22000 on <this-array> and length 1375 on 'y'

Where the contens of test_data/mtsat2_hrit are:

HRIT_MTSAT2_20140524_0001_DK03IR1
HRIT_MTSAT2_20140524_0032_DK01IR1
HRIT_MTSAT2_20140524_0101_DK02IR1
HRIT_MTSAT2_20140524_0601_DK03IR1
HRIT_MTSAT2_20140524_0632_DK01IR1
HRIT_MTSAT2_20140524_0701_DK02IR1
HRIT_MTSAT2_20140524_1201_DK03IR1
HRIT_MTSAT2_20140524_1232_DK01IR1
HRIT_MTSAT2_20140524_1301_DK02IR1
HRIT_MTSAT2_20140524_1801_DK03IR1
HRIT_MTSAT2_20140524_1832_DK01IR1
HRIT_MTSAT2_20140524_1901_DK02IR1

Edit: This isn't supposed to work.

I tried to fix 1 and 2, but I don't know how to handle 3. Any ideas @pnuu?

Tests added and test suite added to parent suite
Tests passed
Passes flake8 satpy

- Move expected_segments to file_types section in YAML - Make reader recognize segment number from the filename

- Separate segmented and non-segmented file types in YAML - Handle missing 'segment' field in the filename

djhoese

LGTM

coveralls · 2019-12-19T16:52:45Z

Coverage increased (+0.0004%) to 87.371% when pulling 004a119 on sfinkens:fix-geo-padding into 8be2ff1 on pytroll:master.

codecov · 2019-12-19T16:52:55Z

Codecov Report

Merging #1021 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1021      +/-   ##
==========================================
+ Coverage   87.37%   87.37%   +<.01%     
==========================================
  Files         183      183              
  Lines       28181    28182       +1     
==========================================
+ Hits        24622    24623       +1     
  Misses       3559     3559

Impacted Files	Coverage Δ
satpy/readers/ahi_hsd.py	`96.58% <100%> (ø)`	⬆️
satpy/tests/reader_tests/test_ahi_hsd.py	`97.94% <100%> (+0.01%)`	⬆️
satpy/readers/yaml_reader.py	`94.17% <100%> (ø)`	⬆️
satpy/readers/olci_nc.py	`90.9% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8be2ff1...004a119. Read the comment docs.

sfinkens added 2 commits December 17, 2019 14:25

Fix padding of ahi_hsd

070496e

- Move expected_segments to file_types section in YAML - Make reader recognize segment number from the filename

Fix padding of mtsat2-imager_hrit

34c38fe

- Separate segmented and non-segmented file types in YAML - Handle missing 'segment' field in the filename

sfinkens requested a review from pnuu December 17, 2019 14:49

mraspaud added bug component:readers labels Dec 19, 2019

Fix ahi_hsd tests

004a119

sfinkens marked this pull request as ready for review December 19, 2019 14:29

sfinkens requested review from djhoese and mraspaud as code owners December 19, 2019 14:29

djhoese approved these changes Dec 19, 2019

View reviewed changes

mraspaud merged commit 4e7b0a2 into pytroll:master Dec 21, 2019

sfinkens deleted the fix-geo-padding branch January 7, 2020 13:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix padding of segmented geostationary images #1021

Fix padding of segmented geostationary images #1021

sfinkens commented Dec 17, 2019 •

edited

djhoese left a comment

coveralls commented Dec 19, 2019

codecov bot commented Dec 19, 2019 •

edited

Fix padding of segmented geostationary images #1021

Fix padding of segmented geostationary images #1021

Conversation

sfinkens commented Dec 17, 2019 • edited

djhoese left a comment

Choose a reason for hiding this comment

coveralls commented Dec 19, 2019

codecov bot commented Dec 19, 2019 • edited

Codecov Report

sfinkens commented Dec 17, 2019 •

edited

codecov bot commented Dec 19, 2019 •

edited