Griddap params

[flackdl]

Hey there,

I'm hoping to use your nifty package but am running into an issue. You should note I don't have a climate science background.

When I run the following I receive an error, which, it seems like the generated URL doesn't conform to the griddap documentation but instead to tabledap.

from erddapy import ERDDAP

e = ERDDAP(
    server='http://oos.soest.hawaii.edu/erddap',
    protocol='griddap',
    response='csv',
    dataset_id='NCEP_Global_Best',
    constraints={
        'time>=': '2018-02-08',
    },
    variables=['time', 'longitude', 'latitude'],
)

# http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time,longitude,latitude&time%3E=1518048000.0
print(e.get_download_url())

Outputs:

Traceback (most recent call last):
  File "/tmp/a.py", line 14, in <module>
    print(e.get_download_url())
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/erddapy.py", line 143, in get_download_url
    constraints=constraints,
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/url_builder.py", line 183, in download_url
    return _check_url_response(url)
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/erddapy/utilities.py", line 101, in _check_url_response
    r.raise_for_status()
  File "/home/danny/.virtualenvs/cwwed-env/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time,longitude,latitude&time%3E=1518048000.0

If I use the Data Access Form and only add the time constraint, it generates a url like:

http://oos.soest.hawaii.edu/erddap/griddap/NCEP_Global_Best.csv?time[(2018-02-01):1:(2018-02-13T12:00:00Z)] which works.

If you browse to the malformed URL it warns about "graphical" params.

I must be doing something wrong but I can't identify what it is. Any help would be appreciated.

[ocefpaf]

Thanks for the report @flackdl! We never really tested erddapy with griddap. I'm flagging this as a bug and we'll try to create a few examples and tests for it in the next version.

For now all I can say is that griddap is not officially supported.

[flackdl]

Ok, thanks for the clarification. It looks like pydap doesn't seem to have a way to programmatically build griddap queries either. I guess griddap and tabledap are opendap extensions so maybe it's not so odd.

[kstonekuan]

Hi, I just wanted to find out more about the difference between tabledap and griddap.

From my understanding of the documentation so far, I believe that while tabledap uses a list of constraints that may work on a number of variables, griddap instead requires each variable to be specified along with all of its dimensions in the correct order (without names). It seems this might not be very compatible with the current implementation of constraints parsing.

Would we need to add a new dimensions attribute to the ERDDAP class and somehow guide users to retrieving the dimension info for each variable in this case?

[avishmehta68710]

we would also have to specify the user in what dimensions he must enter the values

[ocefpaf]

@kstonekuan and @avishmehta68710 I'll add more info soon, been under the weather lately, but look for inspiration in the R erddap package.

[kthyng]

I was stuck on this too — I was under the impression the package worked for griddap since you can input it under protocol. But, the following works for me as a workaround. It essentially just finds the base link to connect to the dataset over opendap and then uses xarray to do the subsetting. This should not be reading in all the available data since it is read in lazily, so I would think it isn't too bad to do it this way.

import xarray as xr
from erddapy import ERDDAP

kw = {
    "min_lon": -99.0,
    "max_lon": -88.0,
    "min_lat": 20.0,
    "max_lat": 30.0,
    "min_time": "2016-07-10T00:00:00",
    "max_time": "2017-02-10T00:00:00",
}

e = ERDDAP(server='https://coastwatch.pfeg.noaa.gov/erddap/', protocol='griddap')
e.dataset_id = 'ucsdHfrE1'
url = e.get_download_url(response='opendap')
xr.open_dataset(url).sel(latitude=slice(kw['min_lat'],kw['max_lat']), longitude=slice(kw['min_lon'],kw['max_lon']), time=slice(kw['min_time'],kw['max_time']))

[ocefpaf]

I was stuck on this too — I was under the impression the package worked for griddap since you can input it under protocol.

That is b/c the we have "some" griddap support. We allow full URLs but not slices at the moment.

But, the following works for me as a workaround. It essentially just finds the base link to connect to the dataset over opendap and then uses xarray to do the subsetting. This should not be reading in all the available data since it is read in lazily, so I would think it isn't too bad to do it this way.

You won't get ERDDAP's server side slicing but to be honest I'm not sure how much better that is when compared to lazy loading. It is ultimately a trade-off between knowing the coords before hand and send your slice request to the server or downloading the coords and then slicing lazily.

[kthyng]

Ok that makes sense. Thank you!

[ocefpaf]

This was implemented during GSoC.

[kthyng]

@ocefpaf cool! Can I look somewhere for more information?

[ocefpaf]

@ocefpaf cool! Can I look somewhere for more information?

https://ioos.github.io/erddapy/01a-griddap-output.html