On some https sites, urllib3/requests latest github version hangs #167

Closed
pythonmobile opened this Issue Mar 25, 2013 · 30 comments

Comments

Projects
None yet
6 participants

I installed Commit 7f9d8ee on python 27 and then tried some https sites with SNI. It seems to be hanging. Is SNI fixed on python2 yet, or was it only for python3?

Owner

shazow commented Mar 25, 2013

Can you include an example of how you're running it? (Note that SNI support requires injecting a contrib module right now.)

Can you please point me to documentation for how to use SNI on python 2. I was not using the contrib module, and hence the problem I think.

Contributor

t-8ch commented Mar 25, 2013

Install:

In your code:

import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()

# [your regular code]

(Have a look at https://github.com/shazow/urllib3/blob/master/urllib3/contrib/pyopenssl.py)

Do I need to do something special to get urllib3.contrib? I tried both pip install urllib3 and sudo pip install git+https://github.com/shazow/urllib3.git -- both don't have 'urllib3.contrib'

Collaborator

sigmavirus24 commented Mar 25, 2013

The setup.py needs to include 'urllib3.contrib' as part of the packages argument to setup

Thanks. Now I could get to an online https page hosted on an SNI server. But this hangs the code:

import urllib3
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
http = urllib3.PoolManager()
r = http.request('GET', 'https://nonexistent.blib.us')

Are there plans to add contrib in setup.packages so that its available by default?

Owner

shazow commented Mar 25, 2013

@pythonmobile ^^^ I think that should do the trick. :) Let me know if it's insufficient.

Owner

shazow commented Mar 25, 2013

@t-8ch added a contrib section to the docs, but looks like it's empty. http://urllib3.readthedocs.org/en/latest/contrib.html

A pull request with an explanation and some examples would be appreciated, if anyone is up for it. :)

Contributor

t-8ch commented Mar 25, 2013

[~]$ curl https://nonexistent.blib.us
curl: (35) Unknown SSL protocol error in connection to nonexistent.blib.us:443 
[~]$ gnutls-cli -p 443 nonexistent.blib.us                           
Processed 159 CA certificate(s).
Resolving 'nonexistent.blib.us'...
Connecting to '216.230.230.76:443'...
*** Fatal error: Error in the pull function.
*** Handshake has failed
GnuTLS error: Error in the pull function.
[~]$ openssl s_client -connect nonexistent.blib.us:443               
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 322 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Firefox also failed.
The server seems to be broken.

@shazow Now you confused me. The contrib docs aren't from me. Anyways sphinx tries to import the module and fails due to missing optional dependencies. I don't know if there is a way to specify dependencies for rtd without forcing them on everyone else.

Owner

shazow commented Mar 25, 2013

@t-8ch Whooops, you're right. I meant @kirkeby. :)

Contributor

t-8ch commented Mar 25, 2013

@shazow some projects add a rtd-requirements.txt file, but I can find no reference for using this. So I assume it's is either picked up automatically or more likely a setting for the project admin to configure.
Another one: https://docs.readthedocs.org/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules

@t-8ch Thanks. The server looks broken indeed. But I was hoping to get an exception from urllib3 instead of it hanging. Is that possible?

Owner

shazow commented Mar 25, 2013

I suspect this will involve specifying a timeout... Or maybe it should fail since it's a handshake thing. Hmm.

Contributor

t-8ch commented Mar 25, 2013

I am getting an exception. But as shazow said, set a timeout.

Thanks guys. I just tested it with requests (all tests pass with the latest commit of urllib3), and had no problems. Perhaps time to get this integrated with requests?

Owner

shazow commented Apr 25, 2013

I assume this is resolved? Please reopen otherwise.

@shazow shazow closed this Apr 25, 2013

@shazow seems like this error is back in urllib3 and in requests.

Owner

shazow commented Jun 29, 2014

@pythonmobile Can you elaborate?

@shazow I just tried this code again:

import urllib3
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
http = urllib3.PoolManager()
r = http.request('GET', 'https://nonexistent.blib.us')

requests hangs. So does urllib3 and everything else. Timeout doesn't have any effect. Maybe I am missing something?

Owner

shazow commented Jun 29, 2014

@pythonmobile What version of urllib3? Can you try the master branch?

I'm getting SSLError: [Errno bad handshake] (54, 'ECONNRESET') in urllib3@master, and similar with curl: curl: (35) Server aborted the SSL handshake.

I was trying urllib3-1.8.3.egg-info. Will try the github version now.

On Sat, Jun 28, 2014 at 10:54 PM, Andrey Petrov notifications@github.com
wrote:

@pythonmobile https://github.com/pythonmobile What version of urllib3?
Can you try the master branch?

I'm getting SSLError: Errno bad handshake in
urllib3@master, and similar with curl: curl: (35) Server aborted the SSL
handshake.


Reply to this email directly or view it on GitHub
#167 (comment).

@shazow This might help. I installed a new version using pip install git+https://github.com/shazow/urllib3.git

Also: OpenSSL.version.version == '0.14'
(and curl hangs as well) - but this works:

root@Temp:~# curl https://asdaasda.blib.us --connect-timeout 1
curl: (28) SSL connection timeout

======================= with urllib3 this happens.

>>>
>>>
>>> import urllib3
>>> import urllib3.contrib.pyopenssl
>>> urllib3.contrib.pyopenssl.inject_into_urllib3()
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'https://nonexistent.blib.us')

*hanged* - I need to press Ctrl+C to get out...
♥♥Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 74, in request
    **urlopen_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 158, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 494, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 296, in _make_reques
t
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 962, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 780, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 218, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 254, in ssl_wrap_
socket
    cnx.do_handshake()
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1075, in do_handshake
    result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt
>>>
Contributor

t-8ch commented Jun 29, 2014

@pythonmobile I also get a bad handshake, like @shazow. (Weirdly it has another error number for me).
Could it be, that your ISP has some sort of transparent proxy which is broken?

Collaborator

sigmavirus24 commented Jun 29, 2014

I get exactly the same errors as @shazow with both curl and urllib3. This definitely seems like something with your ISP (or your network) @pythonmobile

Its not my ISP, but it might be either Debian or VBox networking. Thanks
for all the comments. More on reproducing the problem. I have a windows box
in which requests/urllib behave as they should
(requests.exceptions.SSLError: Errno bad handshake. Now, I create a Debian wheezy 7.5 virtual box (fresh as of
today), and then can reproduce the problem inside the virtual machine. The
networking on the VBox is NAT. The error on requests after Ctrl+C is:

File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1075,
in do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt

Can a virtual machine/debian actually hang a ssl socket?

On Sun, Jun 29, 2014 at 9:22 AM, Ian Cordasco notifications@github.com
wrote:

I get exactly the same errors as @shazow https://github.com/shazow with
both curl and urllib3. This definitely seems like something with your ISP
(or your network) @pythonmobile https://github.com/pythonmobile


Reply to this email directly or view it on GitHub
#167 (comment).

k - Looks like I found what fixes the problem. I've Virtualbox running
debian on a windows machine that uses NAT as the network adapter. If I
change it to Bridge, the connection starts acting up sanely. Perhaps this
is a bug in VBox networking.

Thanks.

On Sun, Jun 29, 2014 at 12:09 PM, David Arken mobilebackup77@gmail.com
wrote:

Its not my ISP, but it might be either Debian or VBox networking. Thanks
for all the comments. More on reproducing the problem. I have a windows box
in which requests/urllib behave as they should
(requests.exceptions.SSLError: Errno bad handshake. Now, I create a Debian wheezy 7.5 virtual box (fresh as of
today), and then can reproduce the problem inside the virtual machine. The
networking on the VBox is NAT. The error on requests after Ctrl+C is:

File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1075,
in do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt

Can a virtual machine/debian actually hang a ssl socket?

On Sun, Jun 29, 2014 at 9:22 AM, Ian Cordasco notifications@github.com
wrote:

I get exactly the same errors as @shazow https://github.com/shazow
with both curl and urllib3. This definitely seems like something with your
ISP (or your network) @pythonmobile https://github.com/pythonmobile


Reply to this email directly or view it on GitHub
#167 (comment).

Collaborator

Lukasa commented Jun 29, 2014

If VirtualBox has a crappy NAT, it's possible that it wasn't correctly mapping the ports.

alvare commented Jun 18, 2015

This just happened to me, sadly.
I get the exact same error output when I hit Ctrl-C, too.

In my case it's an AWS instance, trying to send mails with mailgun and sometimes it hangs forever.

For example, it will send 9 mails and hang forever, and then I Ctrl-C, restart and it will send 400 and hang forever.

alvare commented Jun 19, 2015

I'm on python 2.7.6 btw, @pythonmobile what are you using?

Collaborator

sigmavirus24 commented Jun 19, 2015

@alvare this issue is over a year old. Please open a new one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment