implement Python requests for HTTP

[tomkralidis]

cc @kwilcox

To track implementation of Python requests [1] for HTTP support across the codebase

Between GET / POST, authentication, cookies, etc., the requests library [1] would make OWSLib's HTTP request/response patterns minimal, solid, and very easy to

Notes/ideas:

• perhaps a unified owslib.util.http_request function which all downstream code uses? Or just use directly?
• we should maintain http functions like owslib.util.openURL, owslib.util.http_post in owslib.util, as at least stubs (I've seen these in use in downstreams apps)
• functions like owslib.util.bind_url could probably be deleted afterwards

[1] http://docs.python-requests.org

[kwilcox]

FYI - I've started this, should have something to review in a few days

[kwilcox]

Still working on it...

[QuLogic]

@kwilcox Any progress on this work? I feel like this would greatly help with Python 3, since requests is already compatible.

[Jenselme]

@kwilcox any progress on that issue? I have started to look at what could block a version compatible with both python2 and python3 and the use of urllib2 (splitted in urllib.request and urllib.error in python 3) is problematic.

Switching to requests which supports both python2 and 3 with the same interface would greatly help.

[daf]

@Jenselme @QuLogic I'm working on it now. A few more failing tests to work through today (mostly to do with string/bytes handling as that changed from 2).

(I'm @kwilcox's coworker btw)

[daf]

(here's my branch if you want to pull and test locally for now: https://github.com/daf/OWSLib/commits/3.4)

[Jenselme]

Thanks! If you need some help or specific tests, please let me know.

[daf]

Doctests cause some headaches with the differences between 2 and 3. For instance, here's one I haven't solved yet:

________________________________ [doctest] tests/doctests/namespaces.txt ________________________________
040     # 'om300' does not exist as a namespace, so the below will return nothing
041     >>> ns.get_namespace('om300')
042 
043     # CSW 3.1.1 doesn't exist, so the below will return nothing
044     >>> ns.get_versioned_namespace('csw','3.1.1')
045 
046 
047 Invalid Usage Tests
048 
049     >>> ns.get_namespace()
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,6 @@
     Traceback (most recent call last):
    -...
    -TypeError: get_namespace() takes exactly 2 arguments (1 given)
    +  File "/home/daf/miniconda/envs/owslib3/lib/python3.4/doctest.py", line 1318, in __run
    +    compileflags, 1), test.globs)
    +  File "<doctest namespaces.txt[16]>", line 1, in <module>
    +    ns.get_namespace()
    +TypeError: get_namespace() missing 1 required positional argument: 'key'

The error text changed (much more helpful in 3!). I'm learning more about doctests as I go, does anyone have a strategy to address something like this to work with 2 and 3 while not mutilating the intent of the doctest (aka, readable instead of "make it pass").

Does doctest support partial ellipsis? Something like a literal TypeError: ...

[daf]

Wow, turns out you can do partial line ellipsis. Nice.

[tomkralidis]

Go @daf go!

[Jenselme]

For the UNEXPECTED EXCEPTION: ValueError('Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.',) errors, strings must be converted to bytes and files must be opened as binary. With python3, lxml crashes if it gets a string which defines an encoding: http://lxml.de/parsing.html#python-unicode-strings

I should be able to propose a patch soon.

[Jenselme]

NB: it doesn't produce regression with python2 according to my tests.

[Jenselme]

You can see my patches here: daf#1

I dropped from 32 failed tests in python3 to 16.

[daf]

Thanks @Jenselme !