Basic authentication

[mangecoeur]

There should be support for basic auth when requesting datasets

[jrleeman]

Looks like requests should be able to handle this in a sane way: http://docs.python-requests.org/en/master/user/authentication/#basic-authentication

[lmadaus]

Was searching for some feature like this in Siphon and came across this feature request/issue. I'd like to use Siphon with the THREDDS servers on the NCAR RDA (e.g. https://rda.ucar.edu/thredds/catalog/aggregations/e/ds627.0/5/catalog.html), but accessing those servers requires you to log in/have some sort of cookies file present that Siphon isn't finding or recognizing. Does this ability actually exist in Siphon to supply credentials or is it still being built/considered?

[jrleeman]

Hi Luke - It's still on the roadmap, but we haven't had a chance to dig in yet. Thoughts @dopplershift ?

[dopplershift]

I'll put this on the list for 0.6, since I don't think it should be that hard (at least for basic authentication) and it's probably one of the biggest outstanding to do items.

@lesserwhirls do you have any experiences from how authentication works for netcdf-Java that would be good to know here? How exactly is RDA authentication working?

[lesserwhirls]

For the RDA, the catalogs themselves do not require authentication. However, any service or html/xml page used by a service (like dataset.xml for NCSS) will require authentication.

When you make a request to the server and the resource requires authentication, it will respond with a

HTTP 401 Unauthorized

and, included in those headers, will be something like:

WWW-Authenticate: Basic

For example:

curl -i "https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2015/20150116/gfs.0p25.2015011600.f006.grib2.ascii?LatLon_Projection"
HTTP/1.1 401 Unauthorized
Date: Mon, 31 Jul 2017 13:45:40 GMT
Server: Apache/2.4.6 (CentOS)
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Cache-Control: private
Expires: Wed, 31 Dec 1969 17:00:00 MST
WWW-Authenticate: Basic realm="THREDDS Data Server"
Content-Language: en
Content-Length: 994
Content-Type: text/html;charset=utf-8

So, if you get a 401, you check the WWW-Authenticate part of the header to see what type of authorization is required. Common possibilities include Basic, Digest, and OAuth, although there are others.

Once you get the 401, it's up to the client to decide what to do. In toolsUI, a log-in gui widget will appear:

but there are any number of ways of doing things (e.g. command line prompt, let user set credentials as part of Siphon API).

Now, where this gets a little tricky is when Siphon piggybacks on another library for access, like how we use netCDF4-Python to speak OPeNDAP (which then relies on netCDF-C). The C library does this for auth. Pretty gnarly, but we might be able to get away with detecting the case of a 401 trying to access an OPeNDAP resource, and insert the credentials into the DAP URL (case 2 in the previously linked netCDF-C doc on authentication). However, if the group running the OPeNDAP enabled server has their own authentication service and it involves a redirect (like NASA URS), then things really get fun (see section 4 of the netCDF-C doc), but thankfully NASA has some docs to help guide us for that.

[grisaitis]

What does the community think the interface should be for specifying credentials? As a parameter to TDSCatalog, or any HTTPEndPoint? As environment variables to be picked up in the background?

I’m running into the exact situation mentioned by @lesserwhirls when accessing the RDA: when NCSS is initialized, get_metadata is called, which then does a get request for dataset.xml. Ideally I’d patch the session object but that’s created right before, also during the init for NCSS (subclass of HTTPEndPoint).

So for now, it looks like I’ll have to monkey patch create_http_session so that it returns a requests.Session with something like s = requests.Session(); s.auth = ('user', 'pass').

If anyone has a better solution for making this work with the RDA, please share. I’m pretty new to siphon and THREDDS in general. My goal is to load historical GFS forecasts using pvlib.

[dopplershift]

So in #242, I have moved the creation of sessions to be managed by the HTTPSessionManager, the instance of which can be imported as http_util.session_manager. You can then call session_manager.set_session_options(auth=('user', 'pass')). This seemed far more pragmatic than adding the option to every HTTP-using interface. Does this work for everyone who chimed in here? Any other ideas?