Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WW3 Stations Spec reader #92

Merged
merged 18 commits into from Aug 28, 2023
Merged

Conversation

mpiannucci
Copy link
Contributor

This PR adds support for reading WW3 Station Spectral Wave Files. These files are distributed with GFS Wave output by NOAA. An example location of these files is here: https://noaa-gfs-bdp-pds.s3.amazonaws.com/index.html#gfs.20230820/18/wave/station/bulls.t18z/ . They are also available from NOMADS and GCP.

Because these files are available from cloud providers, the reader accepts a file like object instead of a filename to parse out the dataset.

Example usage with fsspec:

import wavespectra as ws
import fsspec

fs = fsspec.filesystem("s3", anon=True)
with fs.open('s3://noaa-gfs-bdp-pds/gfs.20230820/18/wave/station/bulls.t18z/gfswave.44097.spec') as file:
    ds = ws.read_ww3_station(file)

ds.spec

@mpiannucci mpiannucci changed the title Add WW3 Stations reader Add WW3 Stations Spec reader Aug 21, 2023
@rafa-guedes
Copy link
Collaborator

This looks great @mpiannucci thanks for this PR!

Minor comments before merging this, shall we also support files specified by strings rather than a file object in case those are available locally? I think it would be nice in fact to set this up more broadly for other file types as well as you suggested in #91. Also, could you please run black in the new code you have added, we use it to ensure all the code reads the same.

@mpiannucci
Copy link
Contributor Author

Thanks for the feedback!

I will make those changes and then take a stab at making the changes to support fsspec more broadly.

@mpiannucci
Copy link
Contributor Author

mpiannucci commented Aug 22, 2023

I took care of the low hanging fruit, but the methods that expect file globs for ascii data are a decent bit harder

dset = xr.open_mfdataset(
filename_or_fileglob, chunks=_chunks, combine="by_coords"
)
except ValueError:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get an IndexError rather than a ValueError when trying to open a file-like object with xr.open_mfdataset - do you get ValueError in wavespectra?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives me a ValueError, saying that it does not know which engine to use because it is just a file like object passed in.

import xarray as xr

with open('./sample_files/swanfile.nc', 'rb') as file:
    xr.open_mfdataset(file, chunks={}, combine="by_coords")

If I specify the engine, that seems to give me a ValueError as well:

with open('./sample_files/swanfile.nc', 'rb') as file:
    xr.open_mfdataset(file, chunks={}, combine="by_coords", engine="netcdf4")

gives:

ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

Switching to h5netcdf also throws a ValueError for me:

ValueError: can't open netCDF4/HDF5 as bytes try passing a path or file-like object

Which doesnt totally make sense, but throwing an error does. I am unable to get an IndexError, can you show me the code to reproduce the IndexError? I'm also happy to just catch any exception, but better to be explicit

@mpiannucci
Copy link
Contributor Author

I am not sure if the error is messed up between versions of xarray or netcdf, but for now I just changed it to Exception. If that unacceptable, please let me know.

Otherwise I think this is as far as I want to refactor for this PR, it doesnt remove any existing functionality, but adds flexibility to many of the readers.

* Split open_netcdf into a different function so it can be reused, only raise ValueError and IndexError

* New read_ascii_or_binary function to read str or fileobj, change input functions to use these generic openers when possible
@rafa-guedes rafa-guedes merged commit 7ce49c6 into wavespectra:master Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants