New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

("bad handshake: SysCallError(-1, 'Unexpected EOF')",) despite using verify=False #4244

Closed
adamwilbert opened this Issue Aug 15, 2017 · 32 comments

Comments

Projects
None yet
7 participants
@adamwilbert
Copy link

adamwilbert commented Aug 15, 2017

Summary.

I am trying to make a request to a private api with a private api with an expired certificate that I do not control.

I am attempting to use verify=False in the request, but continue to get a
("bad handshake: SysCallError(-1, 'Unexpected EOF')",) error.

I have tried using the old 2.11 cipher string, but I still cannot complete the request. I have tried creating a custom adapter as detailed here: https://lukasa.co.uk/2017/02/Configuring_TLS_With_Requests/

I am able to recreate the request locally and inside the container that holds my application, but I am not able to make the request within my application without the bad handshake.

Expected Result

Get the same response as I am able to get with cURL/postman/etc.

Actual Result

("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

Reproduction Steps

import requests
requests.post(url, data=data, headers=headers, verify=False)

Have also tried making get requests here without data just to test, they also fail

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.0.3"
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.9.31-moby",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010006f",
    "version": "16.2.0"
  },
  "requests": {
    "version": "2.18.3"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.21.1"
  },
  "using_pyopenssl": true
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

verify=False only prevents us from validating the certificate. Based on this error we aren't getting that far: the server is shutting down the connection. Is the server in question publicly reachable?

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

Oh, wait, I see that curl works. Can you use curl -v and print the debug output here?

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 16, 2017

Note: Unnecessary use of -X or --request, POST is already inferred.

  • Trying ***...
  • TCP_NODELAY set
  • Connected to *** (***) port 443 (#0)
  • TLS 1.0 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
  • Server certificate: .**.com
  • Server certificate: RapidSSL CA
  • Server certificate: GeoTrust Global CA

POST /***.php HTTP/1.1
Host: ***
User-Agent: curl/7.51.0
Accept: /
cache-control: no-cache
content-type: text/xml;charset=UTF-8
Content-Length: 474

  • upload completely sent off: 474 out of 474 bytes
    < HTTP/1.1 200 OK
    < Content-Length: 877629
    < Content-Type: text/xml; charset=ISO-8859-1
    < Server: Microsoft-IIS/6.0
    < Server: NuSOAP Server v0.9.5
    < X-SOAP-Server: NuSOAP/0.9.5 (1.123)
    < X-Powered-By: ASP.NET
    < Date: Wed, 16 Aug 2017 16:09:32 GMT

Hopefully all you need. Thank you so much!

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

Yeah, so the server definitely wants 3DES. The custom adapter linked above should work: can you show me the code you're using with that adapter?

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 16, 2017

            from requests.adapters import HTTPAdapter
            from requests.packages.urllib3.util.ssl_ import create_urllib3_context

            # This is the 2.11 Requests cipher string, containing 3DES.
            CIPHERS = (
                'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
                'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
                '!eNULL:!MD5'
            )

            class DESAdapter(HTTPAdapter):
                """
                A TransportAdapter that re-enables 3DES support in Requests.
                """
                def init_poolmanager(self, *args, **kwargs):
                    context = create_urllib3_context(ciphers=CIPHERS)
                    kwargs['ssl_context'] = context
                    return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

                def proxy_manager_for(self, *args, **kwargs):
                    context = create_urllib3_context(ciphers=CIPHERS)
                    kwargs['ssl_context'] = context
                    return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

            s = requests.Session()
            s.mount('https://feed.***.com', DESAdapter())
            response = s.post('https://feed.***.com/***_server.php', data=SOAP_REQUEST, headers=headers, verify=False)
            data = {}
            data['details'] = self.parse_data(response.content)
            return data

@adamwilbert adamwilbert reopened this Aug 16, 2017

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

You're missing a slash in your call to mount.

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 16, 2017

Sorry that was a typo in editing on GitHub, same code snippet with appropriate url fails

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

Hrm. What happens with allow_redirects=True?

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 16, 2017

Sorry, allow_redirects=False.

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 16, 2017

same error

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 17, 2017

Are you familiar with wireshark or tcpdump? I'd like to see a packet capture of the curl TLS handshake and the Requests TLS handshake.

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 17, 2017

When made through requests

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:56:08.400983 IP (tos 0x0, ttl 64, id 21907, offset 0, flags [DF], proto TCP (6), length 311)
    *** > ***.443: Flags [P.], cksum 0x378e (incorrect -> 0x8e6e), seq 2978143108:2978143379, ack 452897697, win 229, length 271

When made through curl

18:57:34.570901 IP (tos 0x0, ttl 64, id 21293, offset 0, flags [DF], proto TCP (6), length 314)
    *** >***.443: Flags [P.], cksum 0x3791 (incorrect -> 0x7dd4), seq 3275934:3276208, ack 452881097, win 229, length 274
18:57:34.658120 IP (tos 0x0, ttl 37, id 14376, offset 0, flags [none], proto TCP (6), length 1475)
    *** > ***: Flags [P.], cksum 0x81f7 (correct), seq 1:1436, ack 274, win 65535, length 1435
18:57:34.659577 IP (tos 0x0, ttl 64, id 21295, offset 0, flags [DF], proto TCP (6), length 307)
    *** > ***.443: Flags [P.], cksum 0x378a (incorrect -> 0x15fd), seq 274:541, ack 1436, win 251, length 267

Thank you again!

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 17, 2017

Sorry, I need to see the handshake decoded. I'd need a pcap I can open in Wireshark.

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 17, 2017

Let me know if you need anything else.

Thank you.

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 18, 2017

Ok, so that handshake suggests that 3DES is not actually being offered by Requests. Can you please send me an exact copy of the code that sets up the adapter? Because you're sending some cipher suites Requests doesn't normally (CAMELLIA) and not sending 3DES, which I'd expect.

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 18, 2017

        headers = { 'Content-Type': 'text/xml;charset=UTF-8' }

        try:
            # This is the 2.11 Requests cipher string, containing 3DES.
            CIPHERS = (
                'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
                'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
                '!eNULL:!MD5'
            )

            class DESAdapter(requests.adapters.HTTPAdapter):
                """
                A TransportAdapter that re-enables 3DES support in Requests.
                """
                def init_poolmanager(self, *args, **kwargs):
                    context = requests.packages.urllib3.util.\
                                ssl_.create_urllib3_context(ciphers=CIPHERS)
                    kwargs['ssl_context'] = context
                    return super(DESAdapter, self).init_poolmanager(*args,
                                                                    **kwargs)

                def proxy_manager_for(self, *args, **kwargs):
                    context = requests.packages.urllib3.util.\
                                ssl_.create_urllib3_context(ciphers=CIPHERS)
                    kwargs['ssl_context'] = context
                    return super(DESAdapter, self).proxy_manager_for(*args,
                                                                        **kwargs)

            s = requests.Session()
            s.mount('https://feed.***.com', DESAdapter())
            response = s.post('https://feed.***.com/parkme_server.php',
                                data=SOAP_REQUEST, headers=headers,
                                verify=False)
            data = {}
            data['lot_details'] = self.parse_data(response.content)
            return data
        except Exception as error:
            print(error)

the except is just printing ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 18, 2017

Can you put a print statement in the init_poolmanager function to confirm that it is in fact being called?

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 18, 2017

Tried, it is indeed getting called.

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 18, 2017

Gah, that's infuriating. Can you tell me what the type of ssl_context is please?

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 18, 2017

When set in the adapter <class 'urllib3.contrib.pyopenssl.PyOpenSSLContext'>

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 19, 2017

Aha! Interesting...

Can you try uninstalling PyOpenSSL and see if that fixes the problem? You'll still need the DESAdapter, but I have a theory...

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 21, 2017

After uninstalling PyOpenSSL I get
EOF occurred in violation of protocol (_ssl.c:645)

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 23, 2017

@Lukasa any further info on that theory?

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 25, 2017

Hrm, that is weird. I had a theory that said that we weren't applying the custom cipher suites, but apparently not. Can you tcpdump without PyOpenSSL and send me the pcap so I can see if the cipher suites are getting advertised?

@adamwilbert

This comment has been minimized.

Copy link

adamwilbert commented Aug 25, 2017

Thanks again!

@Lukasa

This comment has been minimized.

Copy link
Member

Lukasa commented Aug 28, 2017

@adamwilbert Ok so I'm now extremely perplexed: all of this seems to be ignoring the cipher suites we set. I have no idea why Camellia keeps popping up, but it does, which is a strong suggestion that we're being totally ignored.

However, the new handshake also includes 3DES, and is still being rejected. This suggests 3DES is not your problem. At this stage that makes it very hard to identify what your problem is, without getting an idea of what the hell the server is up to.

@sigmavirus24

This comment has been minimized.

Copy link
Member

sigmavirus24 commented Nov 25, 2017

There's been no update to this since late August (3 months). I'm closing this for now but if we get more information, we can reopen this to investigate it.

@amirouche

This comment has been minimized.

Copy link

amirouche commented Dec 14, 2017

I have the same issue, here is the curl output:

$ curl -v "https://www.europeandataportal.eu/data/en/api/3/action/package_show?id=cz-cuzk-sm5-rk-tren09"
*   Trying 185.3.44.12...
* TCP_NODELAY set
* Connected to www.europeandataportal.eu (185.3.44.12) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP1.1
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; OU=Gandi Standard SSL; CN=europeandataportal.eu
*  start date: Jan  7 00:00:00 2017 GMT
*  expire date: Jan  7 23:59:59 2018 GMT
*  subjectAltName: host "www.europeandataportal.eu" matched cert's "www.europeandataportal.eu"
*  issuer: C=FR; ST=Paris; L=Paris; O=Gandi; CN=Gandi Standard SSL CA 2
*  SSL certificate verify ok.
> GET /data/en/api/3/action/package_show?id=cz-cuzk-sm5-rk-tren09 HTTP/1.1
> Host: www.europeandataportal.eu
> User-Agent: curl/7.55.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 14 Dec 2017 09:14:02 GMT
< Content-Type: application/json;charset=utf-8
< Content-Length: 18548
< Connection: keep-alive
< Pragma: no-cache
< Cache-Control: no-cache
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
< Access-Control-Allow-Headers: X-CKAN-API-KEY, Content-Type
< X-Cache: BYPASS
< 

That said I can not reproduce all the time with requests.

@amirouche

This comment has been minimized.

Copy link

amirouche commented Dec 14, 2017

Here is the debug output of requests when it happens:

[2017-12-14 08:24:29,176: ERROR/ForkPoolWorker-19] error in requests.get: HTTPSConnectionPool(host='www.europeandataportal.eu', port=443): Max retries exceeded with url: /data/en/api/3/action/package_show?id=cz-cuzk-sm5-rk-mlaz28 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))
@breno601

This comment has been minimized.

Copy link

breno601 commented Jun 11, 2018

@Lukasa @adamwilbert Hi, I'm having the exact same issue. How did you fix this without having to uninstall Pyopenssl?

When I do curl I get "SSL connection using ECDHE-RSA-AES128-GCM-SHA256"

I'm using an older version of Requests (2.0.0). If I update to 2.4.0 the issue is gone, but I'm afraid this could break other things.

Thanks!

@misantroop

This comment has been minimized.

Copy link

misantroop commented Aug 1, 2018

Exactly the same issue here. Site seems to want TLS 1.0 and 3DES: https://www.traders.co.jp
No success using the 3DES adapter or forcing TLS 1.0 in PoolManager.

@samvher

This comment has been minimized.

Copy link

samvher commented Sep 27, 2018

I also have the same issue.

Output of curl -v:

* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256

Error message when I send the request:

requests.exceptions.SSLError: HTTPSConnectionPool(host='echomobile.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

Output of python -m requests.help:

{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": "2.3.1"
  }, 
  "idna": {
    "version": "2.7"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.12"
  }, 
  "platform": {
    "release": "4.16.18-041618-generic", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "1010009f", 
    "version": "18.0.0"
  }, 
  "requests": {
    "version": "2.19.1"
  }, 
  "system_ssl": {
    "version": "1000207f"
  }, 
  "urllib3": {
    "version": "1.23"
  }, 
  "using_pyopenssl": true
}

@requests requests locked as resolved and limited conversation to collaborators Sep 28, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.