Added support for NASA/URS SSO service #11

jgallagher59701 · 2015-04-24T21:56:41Z

Roberto,

I added support for the NASA/URS SSO service (which uses OAuth2 but requires that you login using their login page; it does not support Facebook, Google, ..., logins). Also, In the process I added support for site that use things like LDAP in combination with HTTP/S and Basic Auth. My install_basic_client() function reads URIs, usernames and passwords from a .netrc file as well as taking one set as params to the function. I think this code will actually be used by NASA and some of the folks down under (ANU/NCI).

Also added support for a new constant: pydap.lib.SSL_VALIDATE that can be used with sites that take username:password info but use self-signed certs. By default httplib2 does not work with self signed certs.

Anyway, this is my first attempt at python, I tried to make the comments decent...

Can you merge this?

Thanks and hope all is well!
James

Also added support for a new constant: pydap.lib.SSL_VALIDATE that can be used with sites that take username:password info but use self-signed certs. By default PyDAP does not work with self signed certs.

jgallagher59701 · 2016-08-19T21:29:17Z

I'm getting requests to merge this, but I don't have write access to the repository.

I'm willing to merge it and test the result, but I need the permissions.

Thanks
James

jgallagher59701 · 2016-09-15T19:02:22Z

@lewismc Can you look into merging this? Thanks, James

lewismc · 2016-09-15T19:10:18Z

Hi @jgallagher59701 I don't have write permissions either. We are however in need to getting a release for Pydap which is Python3 and Python2.7 compliant. I'll be more than happy to test this PR out.

jgallagher59701 · 2016-09-15T19:13:00Z

On Sep 15, 2016, at 13:10, Lewis John McGibbney notifications@github.com wrote:

Hi @jgallagher59701 https://github.com/jgallagher59701 I don't have write permissions either. We are however in need to getting a release for Pydap which is Python3 and Python2.7 compliant. I'll be more than happy to test this PR out.

@lewismc Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #11 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AH8OKhFr6gICu_OV1t9Kg38fX2bUltGNks5qqZgagaJpZM4EIP7V.

James Gallagher
jgallagher@opendap.org

jameshiebert · 2016-10-21T19:39:40Z

Hi @jgallagher59701,

I've finally taken a good look at this PR and would be happy to include it in the next release that includes Python3. However, there are a couple modules that have been moved around within Pydap. And there are also some standard library modules that should be brought in from the six library (instead of directly imported) to support the renames. I'm happy to take care of that for you, but I did want to ask whether there are any publicly available auth servers that you can include in the examples and the docs? You use "130.56.244.153" as the auth server, but I haven't been able to connect to it, which makes it a little challenging for me to test the code.

jgallagher59701 · 2016-10-21T22:46:10Z

Hi @jameshiebert,

It was quite some time ago I worked on this, but I think it was probably the https://earthdata.nasa.gov/ site which implements OAuth2 for all of NASA's data. If this doesn't work for you, let me know and I'll sort out just how to test the stuff. Thanks!

lewismc · 2016-10-22T01:21:38Z

@jgallagher59701 is correct, all of the authentication goes through URS which can be seen at https://urs.earthdata.nasa.gov/
In all honesty, when we were at ESIP earlier this year, programmatic authentication via URS came up within a conversation with NASA management so I think possibly connecting with earthdata support (linked to from the bottom of the Webpage linked to from above) would be the best option here.

lewismc · 2016-10-22T01:22:03Z

If you want me to drive this then i would be more than happy. Please let me know I work with Earthdata support quite a bit.

jgallagher59701 · 2016-10-22T05:02:45Z

@lewismc That would be great from my POV.

lewismc · 2016-10-23T17:47:29Z

OK doke. I've created a ticket (will post link here once I get a public link) to request application creation within the URS system. We should hear back reasonably soon.

lewismc · 2016-10-25T18:27:22Z

Hi @jameshiebert I am attempting to test out this patch over the top of your develop branch and struggling. As you said, due to the reorganization and removal of some classes, it is unclear as to where the functionality @jgallagher59701 has proposed for this patch (specifcally pydap/lib.py and pydap/util/http.py) should go. Can you be of some assistance please and provide guidance?
Thanks

jameshiebert · 2016-10-25T19:40:08Z

Sure, I can do a review of the patch and will make comments in line about where things should be moved to.

jameshiebert · 2016-10-25T19:42:24Z

pydap/util/urs.py

+
+"""
+
+import cookielib


This should come from six.moves.http_cookiejar

jameshiebert · 2016-10-25T19:45:01Z

pydap/util/urs.py

+    cj = cookielib.CookieJar()
+
+    # Create the password manager and load with the credentials using 
+    pwMgr = urllib2.HTTPPasswordMgrWithDefaultRealm()


from six.moves.urllib.request import HTTPPasswordMgrWithDefaultRealm

jameshiebert · 2016-10-25T19:46:00Z

pydap/util/urs.py

+    if uri and user and passwd:
+        pwMgr.add_password(None, uri, user, passwd)
+
+    opener = urllib2.build_opener(urllib2.HTTPBasicAuthHandler(pwMgr),


Everything here can come from six.moves.urllib.request

jameshiebert · 2016-10-25T19:53:19Z

pydap/util/http.py

@@ -25,7 +25,8 @@ def request(url):
    """
    h = httplib2.Http(cache=pydap.lib.CACHE,
            timeout=pydap.lib.TIMEOUT,
-            proxy_info=pydap.lib.PROXY)
+            proxy_info=pydap.lib.PROXY,
+            disable_ssl_certificate_validation= not pydap.lib.SSL_VALIDATE) # jhrg 4/21/15


Is this a good idea?

jameshiebert · 2016-10-25T19:57:18Z

pydap/util/http.py

@@ -25,7 +25,8 @@ def request(url):
    """


Roberto moved most of the networking code over to src/pydap/net.py, which basically just defines a function GET() which wraps a webob request. If you want to modify SSL validation or proxy settings, you probably want to do it there, with webob or with httplib (that's essentially the software stack from the top down).

jameshiebert · 2016-10-25T19:59:49Z

pydap/util/urs.py

+        return resp, data
+
+    from pydap.util import http
+    http.request = new_request


Same comment as below (pydap/util/http.py) w.r.t. networking code. If you want to monkeypatch Pydap's client (though there's probably a cleaner way to do it), you would want to do so to pydap.net:GET.

jgallagher59701 · 2016-10-28T03:04:56Z

@lewismc Should I jump in here or are you up for it? As you all can tell, this was my first python code...

lewismc · 2016-10-28T03:17:25Z

I'll submit a PR shortly folks sorry. End of a long week.

lewismc · 2016-10-28T03:19:53Z

@jgallagher59701 one other thing, I registered Pydap as an application over on urs.earthdata.nasa.gov so we 'may' be able to utilize urs authentication within the test suite here.

lewismc · 2016-11-02T15:54:24Z

This branch needs to be rebased off of develop branch. I'll have a crack later.

lewismc · 2016-11-15T20:13:15Z

@jameshiebert do you know if it is possible to implement SSL in Webob? I'm thinking about changing the logic in net.py from Webob to Requests so we can implement it. wdyt? BTW, once this niggle is clarified I will implement the solution and submit the PR to update this issue.

jgallagher59701 · 2016-11-16T23:29:42Z

Hi James,

I found an issue with the URS module when using NASA’s opendap server. I noticed they switched to https instead http and now there is an issue where it adds an ‘&’ to the end of the url such as

https://opendap.cr.usgs.gov/opendap/hyrax/MCD15A2H.006/h10v04.ncml?time[0:1:657]&

I have fixed this issue (red) in your urs module as

def new_request(url):
scheme, netloc, path, query, fragment = urlsplit(url)
url = urlunsplit((
scheme, netloc, path, query, fragment
)).rstrip("?&")

log.debug('Opening %s (install_basic_client)' % url)
r = urllib2.urlopen(url)

resp = r.headers.dict
resp['status'] = str(r.code)
data = r.read()

Sorry I don’t even know where I can commit this since your commit has not been merged yet.

Mitch

Mitchell A. Schull, Ph.D.
Post-Doctoral Associate
UMD/ESSIC/CICS @ NOAA / NESDIS / STAR
5830 University Research Court, Suite 2846
College Park, MD
Tel: 301-683-3558
Email: mitch.schull@noaa.gov

jameshiebert · 2016-11-16T23:47:03Z

@lewismc I'd prefer to keep the code base consistent and just use WebOb, rather than trying to mix WebOb with requests. I do love the requests library, but IIRC it's very client oriented, and doesn't give you easy access to the underlying WSGI objects (which we use a ton in the test suite, so that you don't have to actually run an HTTP server to run the test suite).

You should be able to do HTTPS with Webob pretty easily. It looks something like this:

>>> from webob.request import Request
>>> req = Request.blank('https://github.com/pydap/pydap')
>>> resp = req.get_response()
>>> resp.status
'200 OK'

lewismc · 2016-11-16T23:55:07Z

@jameshiebert ah... OK. I didn't realize that everything was encapsulated like that.

lewismc · 2016-11-19T01:27:19Z

@jgallagher59701 do you have a suitable .nc file hosted on one of the OPeNDAP test servers which requires URS authentication to access? If so then I'll code this into our tests.

lewismc · 2016-11-19T01:29:59Z

New PR at #26, sorry for length of time on this one @jgallagher59701

jgallagher59701 · 2016-11-20T02:51:20Z

@lewismc I don't - NASA may, but I'll have to poke around a bit. That is, they have many files, but we'd like something that won't move.

jgallagher59701 · 2016-11-22T00:33:54Z

Here's info from fan.fang-1@nasa.gov about URLs for testing:

James,

Most of our data present themselves through DAP as netCDF-like. So it is a bit odd…

If the data format before DAP has to be netCDF, we have Merra-2 which is netcdf-4, e.g.

http://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/MERRA2/contents.html

(to download original granules http://goldsmr5.gesdisc.eosdis.nasa.gov/data/s4pa/MERRA2/)

And this one in netcdf-3:

http://hydro1.gesdisc.eosdis.nasa.gov/opendap/LPRM_AMSRE_A_SOILM3.002/contents.html

(http://hydro1.gesdisc.eosdis.nasa.gov/data/s4pa/WAOB/LPRM_AMSRE_A_SOILM3.002/)

Let me know if these do not satisfy.

-Fan

lewismc · 2016-11-22T00:55:26Z

@jgallagher59701 ack, I'll continue work over on the branch at #26

jameshiebert · 2016-12-01T21:08:51Z

This functionality should be addressed by #20 which has been merged into master. If it's not for some reason, feel free to re-open this PR.

jgallagher59701 added 3 commits April 24, 2015 15:43

Added support for URS

705dcd5

Also added support for a new constant: pydap.lib.SSL_VALIDATE that can be used with sites that take username:password info but use self-signed certs. By default PyDAP does not work with self signed certs.

Added tests fr ldap and urs authentication

43e814c

Added a README for the authentication (ldap & urs) mods

b087e66

jgallagher59701 mentioned this pull request Jul 28, 2016

Merge URS authentication branch #20

Closed

jameshiebert requested changes Oct 25, 2016

View reviewed changes

jameshiebert mentioned this pull request Oct 29, 2016

NASA openDAP #19

Closed

jameshiebert mentioned this pull request Nov 3, 2016

Adds Python 3.3, 3.4 and 3.5 to supported versions #9

Merged

jameshiebert closed this Dec 1, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for NASA/URS SSO service #11

Added support for NASA/URS SSO service #11

jgallagher59701 commented Apr 24, 2015

jgallagher59701 commented Aug 19, 2016

jgallagher59701 commented Sep 15, 2016

lewismc commented Sep 15, 2016

jgallagher59701 commented Sep 15, 2016

jameshiebert commented Oct 21, 2016

jgallagher59701 commented Oct 21, 2016

lewismc commented Oct 22, 2016

lewismc commented Oct 22, 2016

jgallagher59701 commented Oct 22, 2016

lewismc commented Oct 23, 2016

lewismc commented Oct 25, 2016

jameshiebert commented Oct 25, 2016

jameshiebert Oct 25, 2016

jameshiebert Oct 25, 2016

jameshiebert Oct 25, 2016

jameshiebert Oct 25, 2016

jameshiebert Oct 25, 2016

jameshiebert Oct 25, 2016 •

edited

jgallagher59701 commented Oct 28, 2016

lewismc commented Oct 28, 2016

lewismc commented Oct 28, 2016

lewismc commented Nov 2, 2016

lewismc commented Nov 15, 2016 •

edited

jgallagher59701 commented Nov 16, 2016

jameshiebert commented Nov 16, 2016

lewismc commented Nov 16, 2016

lewismc commented Nov 19, 2016

lewismc commented Nov 19, 2016

jgallagher59701 commented Nov 20, 2016

jgallagher59701 commented Nov 22, 2016

lewismc commented Nov 22, 2016

jameshiebert commented Dec 1, 2016

Added support for NASA/URS SSO service #11

Added support for NASA/URS SSO service #11

Conversation

jgallagher59701 commented Apr 24, 2015

jgallagher59701 commented Aug 19, 2016

jgallagher59701 commented Sep 15, 2016

lewismc commented Sep 15, 2016

jgallagher59701 commented Sep 15, 2016

jameshiebert commented Oct 21, 2016

jgallagher59701 commented Oct 21, 2016

lewismc commented Oct 22, 2016

lewismc commented Oct 22, 2016

jgallagher59701 commented Oct 22, 2016

lewismc commented Oct 23, 2016

lewismc commented Oct 25, 2016

jameshiebert commented Oct 25, 2016

jameshiebert Oct 25, 2016

Choose a reason for hiding this comment

jameshiebert Oct 25, 2016

Choose a reason for hiding this comment

jameshiebert Oct 25, 2016

Choose a reason for hiding this comment

jameshiebert Oct 25, 2016

Choose a reason for hiding this comment

jameshiebert Oct 25, 2016

Choose a reason for hiding this comment

jameshiebert Oct 25, 2016 • edited

Choose a reason for hiding this comment

jgallagher59701 commented Oct 28, 2016

lewismc commented Oct 28, 2016

lewismc commented Oct 28, 2016

lewismc commented Nov 2, 2016

lewismc commented Nov 15, 2016 • edited

jgallagher59701 commented Nov 16, 2016

jameshiebert commented Nov 16, 2016

lewismc commented Nov 16, 2016

lewismc commented Nov 19, 2016

lewismc commented Nov 19, 2016

jgallagher59701 commented Nov 20, 2016

jgallagher59701 commented Nov 22, 2016

lewismc commented Nov 22, 2016

jameshiebert commented Dec 1, 2016

jameshiebert Oct 25, 2016 •

edited

lewismc commented Nov 15, 2016 •

edited