SSLError "bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],) with self signed certificate

[sg77]

Hello. I'm trying to connect to a server with https. I have the certificate which is a self signed certificate which is being included in the verify parameter but the result is an error 'certificate verify failed' error. I was suspecting it had to do with the certificate being self signed (by Microsoft IIS) but with curl this works.
Thanks in advance!

This is the openssl output:

openssl s_client -showcerts -connect server:44300
CONNECTED(00000003)
depth=0 CN = server
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=server
   i:/CN=server
-----BEGIN CERTIFICATE-----
<certificate data here>
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=server
issuer=/CN=server
---
No client certificate CA names sent
Peer signing digest: SHA1
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1477 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: CC4A000083B1E03B446416C9C0B16CBEAB79949E3CF5C936A309A6F92FA01364
    Session-ID-ctx:
    Master-Key: 798A570B0EC2A0CBB7C4C4DE6167E7579A92239942D869CD794B8BEBEA6EB5E492394634AD32665A8BB829DE1F3858D2
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1510329948
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

Expected Result

I'm expecting, since the certificate is in the verify parameter, that the connection won't fail. If I try the same with curl outside python it works:

curl https://server:44300 --cacert /usr/share/ca-certificates/server.crt
 HTTP/1.1 403 Forbidden
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Length: 1158
Content-Type: text/html
Server: Microsoft-IIS/10.0
X-Frame-Options: SAMEORIGIN
P3P: CP=None
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Headers: X-Requested-With,Content-Type
Access-Control-Allow-Credentials: true
Date: Fri, 10 Nov 2017 16:02:26 GMT

(It fails but not because of certificate issues)

Actual Result

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "xxxxx/TestVirtualEnv/local/lib/python2.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "xxxxx/TestVirtualEnv/local/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "xxxxx/TestVirtualEnv/local/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "xxxxx/TestVirtualEnv/local/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "xxxxx/TestVirtualEnv/local/lib/python2.7/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nlybstqvp4nb75n.code1.emi.philips.com', port=44300): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

Reproduction Steps

import requests
requests.get('https://server:44300', verify='/usr/share/ca-certificates/server.crt')

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.1.3"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.12"
  },
  "platform": {
    "release": "4.10.0-38-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010007f",
    "version": "17.3.0"
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

[sg77]

BTW, this is Ubuntu 16.04

[Lukasa]

This error is almost certainly because the certificate itself is invalid in some way. Can you provide the PEM-encoded certificate itself, please?

[sg77]

Hi. Thanks for the quick answer. This is the certificate:
-----BEGIN CERTIFICATE-----
MIID3TCCAsWgAwIBAgIIU/nMdlbWojMwDQYJKoZIhvcNAQELBQAwMDEuMCwGA1UE
AwwlTkxZQlNUUVZQNE5CNzVOLmNvZGUxLmVtaS5waGlsaXBzLmNvbTAeFw0xNzEx
MTAwMDAwMDBaFw0yNzExMTAwMDAwMDBaMDAxLjAsBgNVBAMMJU5MWUJTVFFWUDRO
Qjc1Ti5jb2RlMS5lbWkucGhpbGlwcy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC3mzWep6k1/FbkzzoyZ8QBy/tE8adfwKvw80zaLL+car1bBZ9U
VIXs4es3babtjD3QJWP5/mwoBdIp8gvQkjA1X7RBNJZXbPz6hGR4eqaeRQLrFV9Y
TtB92MA9CDpCXfalCvzzO1jw3zvP1BHUdnTQEwSnnwtf/Ryaud+e7TDxGq8LThmc
glZgO8d2zaYpIjwWx92bXDF/qlqWBkH5mtKIkWOw6Y71xz0Di62cFrMAPEGBjK3c
szpBa5Ttb9+SFtl16t2xDyCbiPFkoMW/4u3Husy/i18hLhEuQwZMHnWsocm+etZ4
8fDt5Bqhab8zC+LKS+Ll7qZdqMHzobeB6j5JAgMBAAGjgfowgfcwXwYDVR0jBFgw
VoAUGdJ3os9nPtTubuwcy1ugtDMdSMChNKQyMDAxLjAsBgNVBAMMJU5MWUJTVFFW
UDROQjc1Ti5jb2RlMS5lbWkucGhpbGlwcy5jb22CCFP5zHZW1qIzMB0GA1UdDgQW
BBQZ0neiz2c+1O5u7BzLW6C0Mx1IwDAMBgNVHRMBAf8EAjAAMAsGA1UdDwQEAwIE
sDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwOwYDVR0RBDQwMoIlTkxZ
QlNUUVZQNE5CNzVOLmNvZGUxLmVtaS5waGlsaXBzLmNvbYIJbG9jYWxob3N0MA0G
CSqGSIb3DQEBCwUAA4IBAQC0MtflowNB4LTLKD1MW3w0QIY5ale3/sEhNCQgHGN5
iNZJptFuFt5jgPGmFjy0Pb5vLMS/Ag1RF3UgTeZzFnaSgyB4mTnwj1gLdwQidVcr
2SlL7TffCj0m/bYjtNbwExRqXE4pQKb5RKwYwpruZaX/G3oHWOG9+2X9Pw5C42zB
OFE4KvYUwOV+noabXvil8LERIdKYxR/2B6qBiwm47IcioqM07zTYLHJ+WDTEMO2k
Qy51yXwFmeOEr5MIBElYCQ0j2AfI4RCXr+2cyUym7tjEr3/I8EsZ5Crvdf++Gwaz
2A05ScPMr+5yfVXygZCenMTwNAyUY1yN9zVj8/n94Psa
-----END CERTIFICATE-----
I know is not entirely valid. But somehow curl is able to accept it. I was wondering why requests does not do the same. I assumed both use openssl to validate the certificate.

[ashwini-kaklij]

Any updates on this? I am also facing issue with self sign certified sites in python request library.

Getting below error.

requests.get('https://10.10.24.20', verify='/etc/ssl/certs/certSIGN_ROOT_CA.pem')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.10.24.20', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

If i keep verify False it works but i want it with verify=True

[uttampawar]

I didn't see this before so I opened a new issue, sorry about that. But I'm facing same issue. For me 'request' fails even with verify=False.

$ python
Python 2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import requests
requests.get("https://localhost:9000/getcpuinfo", verify=False)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

My local certificate info is,

$ openssl s_client -showcerts -connect localhost:9000
CONNECTED(00000003)
depth=0 CN = localhost
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = localhost
verify return:1
write:errno=0
---
Certificate chain
 0 s:/CN=localhost
   i:/CN=localhost
-----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIJAKATu2AY/QT4MA0GCSqGSIb3DQEBCwUAMBQxEjAQBgNV
BAMMCWxvY2FsaG9zdDAeFw0xNzExMDkyMTQ1NTBaFw0xNzEyMDkyMTQ1NTBaMBQx
EjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAJTPk5nao0wG/EDFGnq7BvXMkEZ5oUVq7PAUxWi+E/byJk924l7Z5kACgWBa
zQL0lLXLpdMk97EFGWMblz5Ehtqh7U8HaE9OZ6x/pesDTka+REnpXecklRrdZHX7
lfFnNIU58grPpB2GyUXrRdOtcPlaKXUo+VTd7PgwMtYVtt8pyTWxSB2MMYrqJGT8
78KX6trRzQLm7tas3U0jD59+R8j7gxU6FyFaNJBrkJ5T9kHGKOsAzSqZdCgRBjl5
i7xcXJfOAAnZ3jhGlY5DQht+HZDHhjkLG9kcZZhFDYteFk8drzbd3lBw96nLq+8A
Sy92FtQL4GiYSwZ0WVAmwmTCGjUCAwEAAaNTMFEwHQYDVR0OBBYEFLYjwGKbcV9h
sYHxe8l9UvXVivByMB8GA1UdIwQYMBaAFLYjwGKbcV9hsYHxe8l9UvXVivByMA8G
A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAHl7okBCJlms+cwfzLhs
sbyyuX2wgngxyvjy497zBmeh1TiueGPhOx9u/sfJSZmoUaeRd/zPGkp2DcPQ+Lo2
EHYrXMPE1Ecgpu/15JZ8jNuE+FwZb9lllULLwzq8pDkdbdsSRltdV/rFlZ2YkscB
c+xvVaCltw5KpKnY6AWHoqwoDcd8TZKzyKXLSuluKbHNC1lvg8cMzs6hFA9P92Ae
9P08AKLAIOGJ7QzRrXQIsAO4p9rHheeZeYQZyNiRrXPQUoWos4+OjynaNs+FabhN
XBtSl/GGPRRRfU/D9v4iKfQx15CEvs1AKn1Z6mIPF05pSqbgIoz2mJBV6UM7e+hz
TRs=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=localhost
issuer=/CN=localhost
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1198 bytes and written 302 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 5311B8500C8AF327083E1465FE1E1A6A98E0996B4791150A01D6B130C7F0549909A4BDCDED388E9EDE124BB6C50E150A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1510599077
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
---

[Lukasa]

Please do not hijack this issue. In general you should be using Stack Overflow to ask these questions: if you overload the issue with requests you just make me inclined to close it.

@sg77 Your certificate is marked CA=FALSE, which makes it ineligible to be an issuing certificate. I suspect curl is customising their code to use this certificate as a pin, rather than as a root CA. Requests does not do that: this certificate cannot be a root CA, so it does not validate.

I recommend minting a new certificate with CA=TRUE, or omitting the BasicConstraints altogether.

@ashwini-kaklij I have no idea why your verification is failing, because I can't see the certificate. Please do not post it here: direct your question to StackOverflow instead.

@uttampawar Your error is caused by the server not liking our TLS handshake for some reason. In the absence of more details I cannot determine why. Please, again, take your question to Stack Overflow.

[uttampawar]

@Lukasa I didn't mean to hijack the issue just elevate. May be this is a wrong place. I added my comments and observation since I saw it's of similar nature. I appreciate your feedback. I'll ask on stackoverflow. Thanks.

[popohoma]

Please do not hijack this issue. In general you should be using Stack Overflow to ask these questions: if you overload the issue with requests you just make me inclined to close it.

@sg77 Your certificate is marked CA=FALSE, which makes it ineligible to be an issuing certificate. I suspect curl is customising their code to use this certificate as a pin, rather than as a root CA. Requests does not do that: this certificate cannot be a root CA, so it does not validate.

I recommend minting a new certificate with CA=TRUE, or omitting the BasicConstraints altogether.

@ashwini-kaklij I have no idea why your verification is failing, because I can't see the certificate. Please do not post it here: direct your question to StackOverflow instead.

@uttampawar Your error is caused by the server not liking our TLS handshake for some reason. In the absence of more details I cannot determine why. Please, again, take your question to Stack Overflow.

Hi Lukasa - Just seen your reply. You have replied to @sg77 saying "Your certificate is marked CA=FALSE".

How do you tell this is false and where/how can I set this back to TRUE?

Thanks.

[Synida]

posting the certificate key online is like posting your password

[tedd-wq]

@Synida

posting the certificate key online is like posting your password

Umm, no. Posting the certificate's private key is equivalent to your password.

Every CA in business has their certificate in every mainstream OS and browser. Are they giving their password away to everyone?

https://en.wikipedia.org/wiki/Public-key_cryptography

[OmarHub94]

This error is almost certainly because the certificate itself is invalid in some way. Can you provide the PEM-encoded certificate itself, please?

how can i access to this certificate ?

[RSwarnkar]

If you are beginner to python requests module and want to do some stuffs that requires accessing secure sites, there is high chance that you will be doomed by this error - Certificate verify failed and like all beginner programmers you will be tempted to use auth = session.post( mysecureurl, verify=false)

But this is very bad practice and have been discouraged over a lots of SO posts, but still beginners misuse this because the error is so sucky to fix.
Let me attempt to throw some light on this issue.
Python(pip) and Conda and whatsoever python based software uses separate certificate store just like all browsers do. The Python Requests library uses its own CA file by default, or will use the certifi package's certificate bundle if installed. Also, pip does not use the system certs unlike the curl do.
Hence the for the requests you have to manually specify the certificate store through the conda or pip.

Tldr;

1. Export all the .cer encoded certificate chain using browser as per this amazing blog shown over here. Note, that blog is not about conda certstore but git certstore and it only says to export the root, however I exported all the certificate chains into separate files.
2. Next, install certifi using command pip install certifi
3. Check the default path of the cert store of conda or python:

import ssl
ssl.get_default_verify_paths() OR
import certifi
certifi.where()

4. Once you have located the default cacert.pem file, open this (preferably into Notepad++) and append all the certificate at the end of file. (Take care of certificate demarcation -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----). Save the file and that is it.
   Or if you are using conda use the conda commands:
   conda config --set ssl_verify <pathToYourFile>.crt
   (I have noticed this command updates stuff in C:\Users\johndoe\.condarc)

5. Use the below code to verify:
   import certifi
auth = session.post('https://mysecuresite.com/', cert=());

Also, if you are on linux, you can export custom cacert into system-wide or user profile (.bashrc or .bash_profile) using this link.

[pmullhaupt]

Just in case it helps someone, I was getting the same error and was able to fix it by sending BOTH the verify parameter set to the path of the cert.pem AND the cert parameter set to a tuple of both the cert.pem and key.pem paths.:

    cert_file_path = r'cert.pem'
    key_file_path = r'key.pem'
    cert = (cert_file_path, key_file_path)
    
    
    response_data = requests.post(url, data=data, headers=headers, verify=cert_file_path, cert=cert)

If I only sent the verify parameter or only the cert parameter or verify=True and the cert parameter I would get the error. The above method was the only one that worked.