Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix swath builtin coordinates not being used #1541

Merged
merged 5 commits into from Feb 12, 2021

Conversation

mraspaud
Copy link
Member

@mraspaud mraspaud commented Feb 9, 2021

This PR allows satpy to use longitudes and latitudes provided as coordinates to other dataarrays to create the dataarrays' area attribute.

Consider for example:

<xarray.DataArray '2' (y: 5400, x: 2048)>
dask.array<open_dataset-d4a5694130e597fc2b6cbc2df91fdec72, shape=(5400, 2048), dtype=float64, chunksize=(4096, 2048), chunktype=numpy.ndarray>
Coordinates:
    longitude  (y, x) float64 dask.array<chunksize=(4096, 2048), meta=np.ndarray>
    latitude   (y, x) float64 dask.array<chunksize=(4096, 2048), meta=np.ndarray>
Dimensions without coordinates: y, x
Attributes:
    calibration:    reflectance
    end_time:       2020-01-08 08:34:49.556000
    modifiers:      []
    platform_name:  Metop-C
    resolution:     1050
    sensor:         avhrr-3
    standard_name:  toa_bidirectional_reflectance
    start_time:     2020-01-08 08:19:49.890000
    units:          %
    wavelength:     ['0.725', '0.8625', '1.0', 'µm']

This dataarray contains already populated longitude and latitude coordinates (it is being read from a well-constructed netcdf file), but satpy fails to create the necessary area attribute from it.
This PR fixes it in a manner that uses the builtin coordinates if no other coordinates are provided from eg the dataset definitions of the reader's yaml file.

  • Tests added
  • Passes flake8 satpy
  • Fully documented

@ghost
Copy link

ghost commented Feb 9, 2021

DeepCode failed to analyze this pull request

Something went wrong despite trying multiple times, sorry about that.
Please comment this pull request with "Retry DeepCode" to manually retry, or contact us so that a human can look into the issue.

@TAlonglong
Copy link
Collaborator

Thanks!

My tests with avhrr and viirs data reading back with satpy_cf_nc reader from netcdf files created by satpy cf writer works as expected.

Will take a look at the code later.

Copy link
Collaborator

@TAlonglong TAlonglong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I guess someone as to restart the tests

@@ -823,19 +823,31 @@ def _load_dataset_with_area(self, dsid, coords, **kwargs):
if not file_handlers:
return

area = self._load_dataset_area(dsid, file_handlers, coords, **kwargs)

try:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but how will moving this below affect reading other netcdf data? We rely on the tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do rely on tests yes, and these are failing. I have to take a look at the problem

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see anything wrong with this in principle. You mentioned Satpy wasn't able to make an area from the existing information in the YAML, why not?

@mraspaud
Copy link
Member Author

mraspaud commented Feb 9, 2021

@djhoese because there is no coordinates item defined in the dataset info

@codecov
Copy link

codecov bot commented Feb 11, 2021

Codecov Report

Merging #1541 (c7e597a) into master (5200840) will increase coverage by 0.01%.
The diff coverage is 96.29%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1541      +/-   ##
==========================================
+ Coverage   92.02%   92.03%   +0.01%     
==========================================
  Files         249      249              
  Lines       36309    36384      +75     
==========================================
+ Hits        33413    33486      +73     
- Misses       2896     2898       +2     
Flag Coverage Δ
behaviourtests 4.36% <4.93%> (+0.02%) ⬆️
unittests 92.57% <96.29%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
satpy/tests/test_yaml_reader.py 99.53% <93.54%> (-0.31%) ⬇️
satpy/readers/yaml_reader.py 95.33% <97.36%> (+0.12%) ⬆️
satpy/readers/satpy_cf_nc.py 98.57% <100.00%> (ø)
satpy/tests/reader_tests/test_satpy_cf_nc.py 100.00% <100.00%> (ø)
satpy/readers/nucaps.py 89.14% <0.00%> (-0.05%) ⬇️
satpy/readers/ahi_hsd.py 96.86% <0.00%> (ø)
satpy/readers/ami_l1b.py 96.99% <0.00%> (ø)
satpy/dataset/__init__.py 100.00% <0.00%> (ø)
satpy/writers/__init__.py 88.30% <0.00%> (ø)
satpy/dataset/data_dict.py 90.09% <0.00%> (ø)
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5200840...c7e597a. Read the comment docs.

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To very small things otherwise I think I understand all the yaml reader changes.

lats = coord
if lons is None or lats is None:
raise ValueError('Missing longitude or latitude coordinate: ' + str(coords))
return lats, lons
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick, the return value is the opposite of the method name. Prefer lon, lats I think.

if area is not None:
ds.attrs['area'] = area
ds = add_crs_xy_coords(ds, area)
return ds

@staticmethod
def _assign_builtin_coords(coords, ds):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on changing the name to something like _assign_existing_coords or _assign_coords_from_data_array? To me builtin is ambiguous.

Copy link
Collaborator

@TAlonglong TAlonglong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can not see any problems here. So

LGMT

self.assertTrue(np.all(scn_['image0'].data == self.scene['image0'].data))
self.assertTrue(np.all(scn_['lat'].data == self.scene['lat'].data)) # lat loaded as dataset
self.assertTrue(np.all(scn_['image0'].coords['lon'] == self.scene['lon'].data)) # lon loded as coord
finally:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, never seen a try without an except. So any exceptions raise before finally is kind of ignored with this approach? Instead of

except:
  pass

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the exception isn't ignored, but 'finally' ensures its part of the code is executed before raising the exception.

@mraspaud mraspaud merged commit 55d4d54 into pytroll:master Feb 12, 2021
@mraspaud mraspaud deleted the fix-builtin-coords-swath branch February 12, 2021 07:57
@djhoese djhoese changed the title Fix builtin coords swath Fix swath builtin coordinates not being used Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants