Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort reader table by name + diverse fixes #2745

Merged
merged 8 commits into from
Feb 20, 2024

Conversation

BENR0
Copy link
Collaborator

@BENR0 BENR0 commented Feb 16, 2024

Refactor automatic reader table to be sorted by name by default follwing @djhoese wish (on Slack).

Other changes:

  • Also updated datatable.js to v2 due to deprecation warnings.
  • Re-added info regarding reader status below reader table (probably got lost in some merge) and also added a link to it from the dev guide.
  • Some readers were missing info which led to empty columns (especially the name column) leading to reader appearing at the top of the table. I added missing info to the best of my knowledge but this should probably be checked with the reader authors. Especially the status of said readers and if they support fsspec. Affected readers:

Copy link

codecov bot commented Feb 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (205d5e8) 95.89% compared to head (029c0aa) 95.91%.
Report is 75 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2745      +/-   ##
==========================================
+ Coverage   95.89%   95.91%   +0.02%     
==========================================
  Files         371      373       +2     
  Lines       52825    52967     +142     
==========================================
+ Hits        50656    50803     +147     
+ Misses       2169     2164       -5     
Flag Coverage Δ
behaviourtests 4.14% <ø> (-0.01%) ⬇️
unittests 96.01% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the quick work!

@mraspaud mraspaud added the enhancement code enhancements, features, improvements label Feb 16, 2024
@mraspaud mraspaud added this to the v0.47.0 milestone Feb 16, 2024
@simonrp84
Copy link
Member

Hi,
I'm not sure how to check if my readers support fsspec...which I guess means that they don't support it.

Could you please set the status of my readers to nominal? Thanks!

@BENR0
Copy link
Collaborator Author

BENR0 commented Feb 16, 2024

@simonrp84 will do. Thanks for the feedback.

description: FY-4B AGRI instrument HDF5 reader
status: Beta
supports_fsspec: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonrp84 @BENR0 I could be wrong, but I think any reader that is based on the HDF5 utility file handler now supports fsspec because it uses open_file_or_filename:

try:
f_obj = open_file_or_filename(self.filename)
file_handle = h5py.File(f_obj, "r")

No idea if it performs very well or if h5py works well with it, but I think the support is there.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I think this is true. I have set this to true for those readers using "hdf5_utils" but have not tested this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something that should be tested via unittests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recently added tests for the helper function:

@pytest.mark.parametrize(
("file_thing", "create_read_func"),
[
(lazy_fixture("local_netcdf_filename"), _open_xarray_default),
(lazy_fixture("local_netcdf_filename"), _open_xarray_netcdf4),
(lazy_fixture("local_netcdf_filename"), _open_xarray_h5netcdf),
(lazy_fixture("local_netcdf_path"), _open_xarray_default),
(lazy_fixture("local_netcdf_path"), _open_xarray_netcdf4),
(lazy_fixture("local_netcdf_path"), _open_xarray_h5netcdf),
(lazy_fixture("local_netcdf_fsspec"), _open_xarray_default),
(lazy_fixture("local_netcdf_fsspec"), _open_xarray_h5netcdf),
(lazy_fixture("local_netcdf_fsfile"), _open_xarray_default),
(lazy_fixture("local_netcdf_fsfile"), _open_xarray_h5netcdf),
(lazy_fixture("local_hdf5_filename"), _open_h5py),
(lazy_fixture("local_hdf5_path"), _open_h5py),
(lazy_fixture("local_hdf5_fsspec"), _open_h5py),
],
)
def test_open_file_or_filename(file_thing, create_read_func):
"""Test various combinations of file-like things and opening them with various libraries."""
from satpy.readers import open_file_or_filename
read_func = create_read_func()
open_thing = open_file_or_filename(file_thing)
read_func(open_thing)

@mraspaud
Copy link
Member

Do I understand correctly that this is ready to merge @BENR0 @djhoese ?

@djhoese djhoese merged commit 7c1c92d into pytroll:main Feb 20, 2024
18 of 19 checks passed
@BENR0 BENR0 deleted the fix_reader_table_sort_order branch June 14, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants