Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use lazy loading when response type is OPeNDAP #252

Merged
merged 2 commits into from Jun 14, 2022

Conversation

vinisalazar
Copy link
Contributor

Hi,

this closes #250.

Summary of changes

  • On method ERDDAP.to_xarray, check if response type is opendap. If it is, pass the URL to the xarray.open_dataset method instead of an nc dataset. This allows lazy loading of the dataset.
  • Add corresponding tests (based on the 01a-griddap notebook).

Please feel free to request any changes.

Thank you,
Vini

  - Add fixture 'cswc' (similar to notebook '01a-griddap')
  - Add test 'test_to_xarray_opendap'
  On method ERDDAP.to_xarray, if the response type is opendap, pass the download URL
  to xarray.open_dataset() instead of the nc dataset. This allows using xarray's
  lazy loading capabilities.
@vinisalazar
Copy link
Contributor Author

vinisalazar commented Jun 14, 2022

Not sure what went wrong with the Windows tests... Should we skip this for Windows and macOS like it's done with the iris tests? I'm also wondering why it worked on Py 3.10.

@ocefpaf
Copy link
Member

ocefpaf commented Jun 14, 2022

The failures are happening b/c that ERDDAP server is too restrictive. I need to figure out a way to use VCR there, or change the server, or mock the test. For now, we can just re-start the test and get it working from a fresh GH VM (and new IP).

E   OSError: [Errno -78] NetCDF: Authorization failure: b'https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAquariusSSS3MonthV5'
src\netCDF4\_netCDF4.pyx:1925: OSError
---------------------------- Captured stderr call -----------------------------
syntax error, unexpected $end, expecting ';'
context: Error { code=403; message="Forbidden: Access Forbidden -- Your IP address is on this ERDDAP's request blacklist.Did you often submit more than one request at a time?Did you often submit identical requests in a short period of time?Did you submit a large number of invalid requests?If you are ready to avoid these problems, please email bob.simons@noaa.gov to request to be taken off of the blacklist.";}^

@ocefpaf ocefpaf merged commit e326377 into ioos:main Jun 14, 2022
@ocefpaf ocefpaf added the GSoC22 label Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

griddap opendap request not lazy loading, and failing
3 participants