Host header not being set for CONNECT method of HTTPS proxy

[mctwynne]

Since http.client only supports HTTP 1.0 for the CONNECT method it's not required to send the Host header. Unfortunately, HTTPS proxies often expect a Host header to be present, and fail to connect in its absence. I thought this might be of some interest here since requests + proxies is not particularly uncommon.

Is this something that could be supported in requests?

Expected Result

The expectation is that requests will include the Host header as part of the CONNECTion process.

Actual Result

The Host header is not included.

Reproduction Steps

import requests

>>> requests.get('https://google.ca', timeout=5, verify=False, proxies={'https': 'localhost:8081'})                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                  
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 594, in urlopen                                                                                                  
    self._prepare_proxy(conn)                                                                                                                                                                                       
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy                                                                                           
    conn.connect()                                                                                                                                                                                                  
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect                                                                                                      
    ssl_context=context)                                                                                                                                                                                            
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket                                                                                               
    return context.wrap_socket(sock, server_hostname=server_hostname)                                                                                                                                               
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket                                                                                                                                                        
    _context=self, _session=session)                                                                                                                                                                                
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__                                                                                                                                                           
    self.do_handshake()                                                                                                                                                                                             
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake                                                                                                                                                      
    self._sslobj.do_handshake()                                                                                                                                                                                     
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake                                                                                                                                                       
    self._sslobj.do_handshake()                                                                                                                                                                                     
socket.timeout: _ssl.c:830: The handshake operation timed out                                                                                                                                                       
                                                                                                                                                                                                                    
During handling of the above exception, another exception occurred:                                                                                                                                                 
                                                                                                                                                                                                                    
Traceback (most recent call last):                                                                                                                                                                                  
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send                                                                                                          
    timeout=timeout                                                                                                                                                                                                 
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen                                                                                                  
    _stacktrace=sys.exc_info()[2])                                                                                                                                                                                  
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment                                                                                                    
    raise MaxRetryError(_pool, url, error or ResponseError(cause))                                                                                                                                                  
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.ca', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:830: The handshake operation t
imed out',)))                                                                                                                                                                                                       
                                                                                                                                                                                                                    
During handling of the above exception, another exception occurred:                                                                                                                                                 
                                                                                                                                                                                                                    
Traceback (most recent call last):                                                                                                                                                                                  
  File "<stdin>", line 1, in <module>                                                                                                                                                                               
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/api.py", line 75, in get                                                                                                                 
    return request('get', url, params=params, **kwargs)                                                                                                                                                             
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/api.py", line 60, in request                                                                                                             
    return session.request(method=method, url=url, **kwargs)                                                                                                                                                        
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request                                                                                                       
    resp = self.send(prep, **send_kwargs)                                                                                                                                                                           
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send                                                                                                          
    r = adapter.send(request, **kwargs)                                                                                                                                                                             
  File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/adapters.py", line 510, in send                                                                                                          
    raise ProxyError(e, request=request)                                                                                                                                                                            
requests.exceptions.ProxyError: HTTPSConnectionPool(host='google.ca', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:830: The handshake operation tim
ed out',)))

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.7"
  },
  "platform": {
    "release": "4.15.0-43-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}

[mctwynne]

@kenneth-reitz Do you happen to have a rough idea of how much effort would be involved in supporting this feature?

[mirkokg]

I am also having huge problem with this as well.

All of our proxies require at least HTTP/1.1 👎

Requests tries to connect using HTTP/1.0 and fails with:

RemoteDisconnected('Remote end closed connection without response')

I tried patch client _tunnel function to send 1.1 header and Host, the same way CURL sends, but I am getting Invalid response afterwards on:

response = self.response_class(self.sock, method=self._method)

probably lot more changes are needed on tunneling https requests...

By the way http requests work normally.

[sigmavirus24]

I don't believe this is a bug or feature in requests. Rather, we rely on underlying libraries to generate the appropriate Host header and HTTP version. I know urllib3 just added support for TLS-in-TLS (using TLS to connect to a TLS proxy and sending a TLS request over that). Perhaps things have become better and this is maybe fixed (or at least partially better).

[mirkokg]

Yeah, this is part of client.py: https://raw.githubusercontent.com/python/cpython/master/Lib/http/client.py
I tried latest master branch, and tunneling https over https connect proxy is unfortunately not working.
I also tried this pull request:

python/cpython#8305

but unfortunately that didn't solve the issue.

[sethmlarson]

(Btw urllib3 TLS+TLS support hasn't been merged or released yet, still in development)

[mirkokg]

Thanks sethmlarson, I just tested urllib3 branch, and TLS+TLS works perfectly! Thanks for the tip!

https://github.com/jalopezsilva/urllib3/tree/tls_in_tls_integration