Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADD: Options to select ODIM HDF5 datasets #1020

Merged
merged 1 commit into from Nov 20, 2021

Conversation

ritvje
Copy link
Contributor

@ritvje ritvje commented Nov 19, 2021

Hi! Here's another pull request from my personal modifications. At the Finnish Meteorological Institute, the radar volumes in ODIM HDF5 format store all elevation scans in the same file, but some of the scans have different amount of range bins and different range resolution. This makes it impossible to read the volumes with pyart without picking the correct scans into separate files first with some external tool.

So as a workaround, I'm suggesting adding options include_datasets and exclude_datasets in pyart.aux_io.read_odim_h5 to select the datasets that are read. The default options select all datasets, so old behaviour is retained.

For example (the volume file is available from AWS):

>>> radar = pyart.aux_io.read_odim_h5("202111190500_radar.polar.fivih.h5")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ritvanen/Documents/koodaus/pyart/pyart/aux_io/odim_h5.py", line 231, in read_odim_h5
    raise ValueError('range scale changes between sweeps')
ValueError: range scale changes between sweeps
>>> radar = pyart.aux_io.read_odim_h5("202111190500_radar.polar.fivih.h5", include_datasets=["dataset1", "dataset2"])
>>> print(radar.fixed_angle)
{'long_name': 'Target angle for sweep', 'units': 'degrees', 'standard_name': 'target_fixed_angle', 'data': array([0.3, 0.7], dtype=float32)}

Allow user to select included/excluded datasets in
aux_io.read_odim_hdf5 to allow handling also ODIM HDF5 files
with varying `nbins` between the datasets
@zssherman
Copy link
Collaborator

Reclosing and reopening, for some reason travis did not run.

@zssherman zssherman closed this Nov 19, 2021
@zssherman zssherman reopened this Nov 19, 2021
@zssherman
Copy link
Collaborator

zssherman commented Nov 19, 2021

With the CI fixed, @ritvje I think this is a great fix. I have thought of this before and how I can maybe get these readers to still read these datasets with differing range bins. Might be easier once we start integrating Xarray which hopefully will begin soon.

@ritvje
Copy link
Contributor Author

ritvje commented Nov 20, 2021

Yeah, having different range bins would be great, but in the meanwhile I think this would be a good solution. And even with shared range bins, often reading the whole volume is not needed, so selecting only the required datasets saves also memory.

Is there anything I should do regarding the failed checks? I'm not an expert, but it looks to me like the failed tests don't use this function.

@zssherman
Copy link
Collaborator

@ritvje Looks like there was an update to matplotlib that broke the unittests, yeah unrelated to this PR, so merging.

@zssherman
Copy link
Collaborator

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants