SSL/TLS server certificate validation #1202

patatetom · 2021-11-10T11:14:39Z

Checklist

I've searched for similar issues.
I'm using the latest version of HTTPie.

Context

(Arch)Linux 5.10.78-1-lts
python 3.9.7
httpie 2.6.0
rootCA/interCA/serverCA
interCA system trusted
rootCA system trusted
web browsers (Cr/Ff) OK
curl/wget OK

Minimal reproduction code and steps

https https://server
https --verify rootCA.pem https://server

Current result

https: error: SSLError: HTTPSConnectionPool(host='server', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))) while doing a GET request to URL: https://server/
OK

Expected result

OK
OK

Debug output

https --debug https://server/
HTTPie 2.6.0
Requests 2.26.0
Pygments 2.10.0
Python 3.9.7 (default, Oct 10 2021, 15:13:22) 
[GCC 11.1.0]
/home/patatetom/.local/pipx/venvs/httpie/bin/python
Linux 5.10.78-1-lts

<Environment {'colors': 256,
 'config': {'__meta__': {'about': 'HTTPie configuration file',
                         'help': 'https://httpie.org/doc#config',
                         'httpie': '1.0.2'},
            'default_options': []},
 'config_dir': PosixPath('/home/patatetom/.httpie'),
 'devnull': <property object at 0x7fb864b97090>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x7fb864b84f70>,
 'program_name': 'https',
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

<PluginManager {'adapters': [],
 'auth': [<class 'httpie.plugins.builtin.BasicAuthPlugin'>,
          <class 'httpie.plugins.builtin.DigestAuthPlugin'>],
 'converters': [],
 'formatters': [<class 'httpie.output.formatters.headers.HeadersFormatter'>,
                <class 'httpie.output.formatters.json.JSONFormatter'>,
                <class 'httpie.output.formatters.xml.XMLFormatter'>,
                <class 'httpie.output.formatters.colors.ColorFormatter'>]}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'headers': {'User-Agent': b'HTTPie/2.6.0'},
 'method': 'get',
 'params': <generator object MultiValueOrderedDict.items at 0x7fb864a795f0>,
 'url': 'https://server/'})


https: error: SSLError: HTTPSConnectionPool(host='server', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))) while doing a GET request to URL: https://server/

Traceback (most recent call last):
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='server', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patatetom/.local/bin/https", line 8, in <module>
    sys.exit(main())
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/httpie/__main__.py", line 9, in main
    exit_status = main()
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/httpie/core.py", line 70, in main
    exit_status = program(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/httpie/core.py", line 169, in program
    for message in messages:
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/httpie/client.py", line 102, in collect_messages
    response = requests_session.send(
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/patatetom/.local/pipx/venvs/httpie/lib/python3.9/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='server', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

Additional information, screenshots, or code examples

hi,
it seems that it is the use of the custom SSL context which poses problem because the direct use of the python ssl module does not pose any particular problem to access https://server/...
regards, lacsaP.

The text was updated successfully, but these errors were encountered:

isidentical · 2021-12-28T10:26:39Z

@patatetom can you provide the example website for us to reproduce this problem?

patatetom · 2021-12-28T15:48:11Z

hi @isidentical,
sorry, I can't because it's an internal server, but I can run a series of tests... (well, not right now because I'm on vacation).

patatetom · 2022-01-19T11:10:43Z

hi,

first of all, I didn't manage to simulate the problem :

I mounted two (Arch)Linux virtual machines,
I applied this https://github.com/patatetom/tls4dev/
and httpie behaved like curl and wget, i.e. as expected.

so i rechecked my configuration :

according to openssl, my three certificates are OK

openssl verify -CAfile my.root.cert.pem -untrusted my.intermediate.cert.pem my.server.cert.pem
my.server.cert.pem: OK

they are installed at system level

trust list | egrep -i 'label:…'

curl works as expected

server=…

curl --disable --head https://$server/ && echo OK
HTTP/2 200 
server: nginx
date: Wed, 19 Jan 2022 10:14:49 GMT
content-type: text/html; charset=UTF-8
content-length: 5076
last-modified: Tue, 04 Feb 2020 15:39:52 GMT
etag: "5e399048-13d4"
strict-transport-security: max-age=31536000 ;includeSubDomains; preload
x-frame-options: deny
accept-ranges: bytes
OK

wget also

wget --server-response --output-document=/dev/null --quiet https://$server/ && echo OK
SSL_INIT
  HTTP/1.1 200 OK
  Server: nginx
  Date: Wed, 19 Jan 2022 10:19:57 GMT
  Content-Type: text/html; charset=UTF-8
  Content-Length: 5076
  Last-Modified: Tue, 04 Feb 2020 15:39:52 GMT
  Connection: keep-alive
  ETag: "5e399048-13d4"
  Strict-Transport-Security: max-age=31536000 ;includeSubDomains; preload
  X-Frame-Options: deny
  Accept-Ranges: bytes
OK

httpie meets the error

python -m venv test.httpie

source test.httpie/bin/activate

pip list
Package    Version
---------- -------
pip        21.2.4
setuptools 58.1.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/tmp/ssl/test.httpie/bin/python -m pip install --upgrade pip' command.

pip install --upgrade pip
Requirement already satisfied: pip in ./test.httpie/lib/python3.10/site-packages (21.2.4)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3.1

pip install httpie
Collecting httpie
  Using cached httpie-2.6.0-py3-none-any.whl (63 kB)
Requirement already satisfied: setuptools in ./test.httpie/lib/python3.10/site-packages (from httpie) (58.1.0)
Collecting requests-toolbelt>=0.9.1
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting defusedxml>=0.6.0
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting Pygments>=2.5.2
  Using cached Pygments-2.11.2-py3-none-any.whl (1.1 MB)
Collecting requests[socks]>=2.22.0
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting charset-normalizer>=2.0.0
  Using cached charset_normalizer-2.0.10-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting PySocks!=1.5.7,>=1.5.6
  Using cached PySocks-1.7.1-py3-none-any.whl (16 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, PySocks, requests-toolbelt, Pygments, defusedxml, httpie
Successfully installed PySocks-1.7.1 Pygments-2.11.2 certifi-2021.10.8 charset-normalizer-2.0.10 defusedxml-0.7.1 httpie-2.6.0 idna-3.3 requests-2.27.1 requests-toolbelt-0.9.1 urllib3-1.26.8

pip list
Package            Version
------------------ ---------
certifi            2021.10.8
charset-normalizer 2.0.10
defusedxml         0.7.1
httpie             2.6.0
idna               3.3
pip                21.3.1
Pygments           2.11.2
PySocks            1.7.1
requests           2.27.1
requests-toolbelt  0.9.1
setuptools         58.1.0
urllib3            1.26.8

https --header https://$server/ && echo OK
https: error: SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)'))) while doing a GET request to URL: https://…/

on the python side (in httpie venv), the http.client works

python <<~~~~
import http.client
server="$server"
conn = http.client.HTTPSConnection(server)
conn.request('GET', '/')
resp = conn.getresponse()
print(resp.status)
print(resp.headers.items())
~~~~
200
[('Server', 'nginx'), ('Date', 'Wed, 19 Jan 2022 10:27:52 GMT'), ('Content-Type', 'text/html; charset=UTF-8'), ('Content-Length', '5076'), ('Last-Modified', 'Tue, 04 Feb 2020 15:39:52 GMT'), ('Connection', 'keep-alive'), ('ETag', '"5e399048-13d4"'), ('Strict-Transport-Security', 'max-age=31536000 ;includeSubDomains; preload'), ('X-Frame-Options', 'deny'), ('Accept-Ranges', 'bytes')]

on the other hand requests module (always in httpie venv) meets the error

python <<~~~~
import requests
req = requests.get('https://$server/')
print(req.status_code)
~~~~
Traceback (most recent call last):
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/ssl/test.httpie/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

I hope all this can help...
I remain available for other tests ;-)
regards, lacsaP.

patatetom · 2022-01-19T11:31:06Z

some additional tests for httpie :

https --verify=my.root.cert.pem --header https://$server/ && echo OK
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 5076
Content-Type: text/html; charset=UTF-8
Date: Wed, 19 Jan 2022 11:23:28 GMT
ETag: "5e399048-13d4"
Last-Modified: Tue, 04 Feb 2020 15:39:52 GMT
Server: nginx
Strict-Transport-Security: max-age=31536000 ;includeSubDomains; preload
X-Frame-Options: deny
OK

https --ssl=ssl2.3 --header https://$server/ && echo OK
https: error: SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)'))) while doing a GET request to URL: https://…/

https --ssl=tls1 --header https://$server/ && echo OK
https: error: SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)'))) while doing a GET request to URL: https://…/

https --ssl=tls1.1 --header https://$server/ && echo OK
https: error: SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)'))) while doing a GET request to URL: https://…/


https --ssl=tls1.2 --header https://$server/ && echo OK
https: error: SSLError: HTTPSConnectionPool(host='…', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)'))) while doing a GET request to URL: https://…/

patatetom added bug Something isn't working new Needs triage. Comments are welcome! labels Nov 10, 2021

isidentical changed the title ~~SSL/TLS server certificat validation~~ SSL/TLS server certificate validation Dec 28, 2021

isidentical added awaiting-response and removed new Needs triage. Comments are welcome! labels Dec 28, 2021

Ousret linked a pull request Dec 30, 2023 that will close this issue

Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) #1531

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSL/TLS server certificate validation #1202

SSL/TLS server certificate validation #1202

patatetom commented Nov 10, 2021

isidentical commented Dec 28, 2021

patatetom commented Dec 28, 2021

patatetom commented Jan 19, 2022 •

edited

patatetom commented Jan 19, 2022

SSL/TLS server certificate validation #1202

SSL/TLS server certificate validation #1202

Comments

patatetom commented Nov 10, 2021

Checklist

Context

Minimal reproduction code and steps

Current result

Expected result

Debug output

Additional information, screenshots, or code examples

isidentical commented Dec 28, 2021

patatetom commented Dec 28, 2021

patatetom commented Jan 19, 2022 • edited

patatetom commented Jan 19, 2022

patatetom commented Jan 19, 2022 •

edited