Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fsspec support to li_l2_nc reader #2753

Merged
merged 6 commits into from Mar 8, 2024
Merged

Conversation

pnuu
Copy link
Member

@pnuu pnuu commented Mar 7, 2024

This PR adds remote reading support to li_l2_nc reader via fsspec.

The change in NetCDF4FileHandler.collect_cache_vars() for getting the attributes is that for h5netcdf backend the v.__dict__ returns (for longitude dataset as an example):

{'_parent_ref': <weakref at 0x7fb98d573420; to 'File' at 0x7fb98d1d54f0>, '_root_ref': <weakref at 0x7fb98d573420; to 'File' at 0x7fb98d1d54f0>, '_h5path': '/longitude', '_dimensions': ('flashes',), '_initialized': True}

instead of v.attrs

<class 'h5netcdf.attrs.Attributes'>
_FillValue: -32767
long_name: 'Longitude of flash'
units: 'degrees_east'
standard_name: 'longitude'
scale_factor: 0.0027
add_offset: 0.0

This could be wrapped in dict(v.attrs) but apparently this already work as dictionary so there's no need.

The change is needed for the coordinates to be handled properly by setting standard_name, scale and offset.

  • Closes #xxxx
  • Tests added
  • Fully documented
  • Add your name to AUTHORS.md if not there already

@pnuu pnuu added enhancement code enhancements, features, improvements component:readers labels Mar 7, 2024
@pnuu pnuu self-assigned this Mar 7, 2024
Copy link

codecov bot commented Mar 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.93%. Comparing base (ed751f8) to head (12854de).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2753   +/-   ##
=======================================
  Coverage   95.92%   95.93%           
=======================================
  Files         375      375           
  Lines       53167    53237   +70     
=======================================
+ Hits        51003    51075   +72     
+ Misses       2164     2162    -2     
Flag Coverage Δ
behaviourtests 4.12% <0.00%> (-0.01%) ⬇️
unittests 96.03% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines 261 to 264
try:
attrs = v.attrs
except AttributeError:
attrs = v.__dict__
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very familiar with this, but it feels like it deserves a test...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but I have no idea how to test. We don't access any real files in the tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we then should add one test with a real file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have an idea, lets see if I can complete it today.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f1bc699 should cover the addition. Fails with the original and passes with the current implementation for h5netcdf backend (remote reading).

try:
arr = xr.DataArray(
variable[:], dims=variable.dimensions, attrs=attrs, name=variable.name)
except ValueError:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, does the ValueError come from the variable[:]? Would it make more sense to have a try/except dedicated to just the array creation and then a single xr.DataArray(...) creation at the end (that uses the attrs and var_arr or whatever created from the previous try/excepts).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's it. I refactored as suggested and added a test for the scalar case in 12854de

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mraspaud mraspaud merged commit 30cce11 into pytroll:main Mar 8, 2024
18 of 19 checks passed
@pnuu pnuu deleted the feature-li-fsspec branch March 8, 2024 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants