urllib2: urlopen() gets exception(kwargs bug?)

[Douman]

BPO	23245
Nosy	@tiran, @alex, @bitdancer, @demianbrecht, @spaceone
Files	callstack_urllib2: Callstack callstack_urllib2_full: full callstack

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-02-24.15:50:41.798>
created_at = <Date 2015-01-15.06:01:34.455>
labels = ['invalid', 'type-bug', 'library']
title = 'urllib2: urlopen() gets exception(kwargs bug?)'
updated_at = <Date 2016-09-26.14:54:06.352>
user = 'https://bugs.python.org/Douman'

bugs.python.org fields:

activity = <Date 2016-09-26.14:54:06.352>
actor = 'spaceone'
assignee = 'none'
closed = True
closed_date = <Date 2015-02-24.15:50:41.798>
closer = 'Douman'
components = ['Library (Lib)']
creation = <Date 2015-01-15.06:01:34.455>
creator = 'Douman'
dependencies = []
files = ['37713', '37715']
hgrepos = []
issue_num = 23245
keywords = []
message_count = 13.0
messages = ['234055', '234073', '234082', '234083', '234084', '234085', '234088', '236433', '236508', '277421', '277422', '277425', '277426']
nosy_count = 6.0
nosy_names = ['christian.heimes', 'alex', 'r.david.murray', 'demian.brecht', 'Douman', 'spaceone']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23245'
versions = ['Python 2.7']

[Douman]

I get strange callstack from urllib2

It seems that python thinks that HTTPSConnection doesn't have context argument.
Which is entirely incorrect.
I have suspicions that this is related to the way how context argument is passed(kwargs)
This happens starting from python 2.7.9

[bitdancer]

More likely http_class isn't what you think it is (ie: it might be an HTTPSConnection subclass that hasn't been updated to deal with 2.7.9. Can you check that?

[Douman]

Yes, i checked what is http_class. It is passed as httplib.HTTPSConnection
Before submitting this issue i checked httplib.py in my installation of py2 and there https://hg.python.org/cpython/file/2.7/Lib/httplib.py

HTTPSConnection has argument "context".

Btw, it would be nice to update comments in urllib2 so that it would be more accurate

[Douman]

Btw, i also tried to replace **kwargs with usual argument and it didn't throw exception

[Douman]

Also according to documentation this class was specifically updated with context in 2.7.9
I guess then there should commit related to adding of "context" to HTTPSConnection

[Douman]

It seems to be this one
https://hg.python.org/cpython/rev/1882157b298a

[Douman]

I made additional experiments and now i'm confused
When i tried to execute urlopen in python interpeter it worked fine.
But it fails for me when i attempt to do so via some helper function in search engine of qBittorent it throw exception

https://github.com/qbittorrent/qBittorrent/blob/master/src/searchengine/nova/helpers.py

What is curious... all exceptions should be passed without notices, but for some reason python does throw exception even though all are catched and passed

[Douman]

Just to up issue.
It seems that there is some changes in 2.7.9 that breaks usage of urllib2.urlopen()

[Douman]

No problem in python implementation of HTTPSConnection
Sorry

[spaceone]

Hello,
The resolution of this bug is "not a bug". If that is the case can you please add information how to fix/workaround this.
I have got the following valid-seeming code:
"""
import cookielib
import urllib
import urllib2
cookie = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie))
opener.open('https://www.google.com', timeout=2)
"""

Which results in:
Traceback (most recent call last):
  File "httplib_context_bug.py", line 6, in <module>
    opener.open('https://www.google.com', timeout=2)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1166, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
TypeError: __init__() got an unexpected keyword argument 'context'

$ python 
Python 2.7.9 (default, Mar  1 2015, 12:57:24)
$ cat /etc/issue
Debian GNU/Linux 8 \n \l

[tiran]

You get advice on the Python user mailing list, https://mail.python.org/mailman/listinfo/python-list . The bug tracker is not a support/help forum.

[bitdancer]

Christian: it looks like, unlike the original report, this one involves only stdlib code. So maybe there really is a bug here.

[spaceone]

Nice that you investigate again into this issue. Could you please test if this still happens on python 2.7.10 as I unfortunately have no environment with that version. This would be very kind of you!