TypeError: startswith first arg must be bytes or a tuple of bytes, not str. Python 3.

[michaelhelmick]

Python 3

requirements.txt

requests==1.2.0
requests_oauthlib==0.3.1

Stacktrace:

Traceback (most recent call last):
  File "/home/travis/build/ryanmcgrath/twython/test_twython.py", line 293, in test_create_and_destroy_favorite
    self.api.create_favorite(id=test_tweet_id)
  File "/home/travis/build/ryanmcgrath/twython/twython/twython.py", line 87, in <lambda>
    return lambda **kwargs: self._constructFunc(key, deprecated_key, **kwargs)
  File "/home/travis/build/ryanmcgrath/twython/twython/twython.py", line 126, in _constructFunc
    content = self._request(url, method=fn['method'], params=kwargs)
  File "/home/travis/build/ryanmcgrath/twython/twython/twython.py", line 142, in _request
    response = func(url, data=params, files=files)
  File "/home/travis/virtualenv/python3.3/lib/python3.3/site-packages/requests/sessions.py", line 399, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/home/travis/virtualenv/python3.3/lib/python3.3/site-packages/requests/sessions.py", line 354, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.3/lib/python3.3/site-packages/requests/sessions.py", line 455, in send
    adapter = self.get_adapter(url=request.url)
  File "/home/travis/virtualenv/python3.3/lib/python3.3/site-packages/requests/sessions.py", line 489, in get_adapter
    if url.startswith(prefix):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Alright, so basically, my tests are failing.
It ends up being that when using oauthlib (from requests-oauthlib) in conjunction with requests.. oauthlib actually converts the url being passed to bytes and when requests/sessions.py tries to get the adapter on line 455 (requests 1.2.0) in the get_adapter function, it goes through a forloop and at that point (line 489):

 if url.startswith(prefix):

url is bytes from oauthlib.oauth1.Client.sign every point before that it's a str (trust me, I did extensive filtering to see where the hell it was getting converted from str to bytes haha)

Where should this be fixed though?

• oauthlib.. it should be returning str for the url in the sign function
• requests_oauthlib.. after it calls self.client.sign from oauthlib.oauth1.Client.sign we should convert r.url back to a str before sending it to requests
  OR
• requests.. always convert url to a str in get_adapter

[michaelhelmick]

/cc @idan

[michaelhelmick]

Specifically, in oauthlib it happens here: https://github.com/idan/oauthlib/blob/master/oauthlib/oauth1/rfc5849/__init__.py#L288-L290
Before the URI is encoded, it's a str and after.. bytes

Also, via requests_oauthlib passing None for the decoding kwarg, this error is raised:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./twython/twython.py", line 87, in <lambda>
    return lambda **kwargs: self._constructFunc(key, deprecated_key, **kwargs)
  File "./twython/twython.py", line 127, in _constructFunc
    content = self._request(url, method=fn['method'], params=kwargs)
  File "./twython/twython.py", line 144, in _request
    response = func(url, params=params)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/sessions.py", line 372, in get
    return self.request('GET', url, **kwargs)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/sessions.py", line 464, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/adapters.py", line 256, in send
    r = self.build_response(request, resp)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/adapters.py", line 125, in build_response
    extract_cookies_to_jar(response.cookies, req, resp)
  File "/Users/mikehelmick/.virtualenv/twython3k/lib/python3.3/site-packages/requests/cookies.py", line 105, in extract_cookies_to_jar
    jar.extract_cookies(res, req)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/cookiejar.py", line 1647, in extract_cookies
    if self._policy.set_ok(cookie, request):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/cookiejar.py", line 931, in set_ok
    if not fn(cookie, request):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/cookiejar.py", line 952, in set_ok_verifiability
    if request.unverifiable and is_third_party(request):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/cookiejar.py", line 707, in is_third_party
    if not domain_match(req_host, reach(request.origin_req_host)):
AttributeError: 'MockRequest' object has no attribute 'origin_req_host'

[Lukasa]

oauthlib isn't liable here: requests-oauthlib explicitly asks for bytestrings back. I think requests-oauthlib needs to sort this out. I'll take a look.

[Lukasa]

To explain, we ask for bytes back from oauthlib in requests-oauthlib because of the various UnicodeDecodeErrors we got in #1252.

[Lukasa]

See requests/requests-oauthlib#43 for the fix.

[Lukasa]

Resolved by requests/requests-oauthlib#43. As always Mike, thanks for the bug report and the investigation. =)