HTTP only system proxy not working properly on Windows

[kotori2]

Subject

Since Python 3.9, both requests and pip not working properly with a Windows system proxy set.

Environment

Describe your environment.
At least, paste here the output of:

>>> print("OS", platform.platform())
OS Windows-10-10.0.19041-SP0
>>> print("Python", platform.python_version())
Python 3.9.1
>>> print("urllib3", urllib3.__version__)
urllib3 1.26.2

PS > Write-Output $env:https_proxy
PS > Write-Output $env:http_proxy
PS > Get-ItemProperty -Path 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings' | findstr ProxyServer
ProxyServer              : 127.0.0.1:7890

Steps to Reproduce

1. Set Windows proxy from Setting - Network and Internet - Proxy. This proxy should support HTTP only.

import requests
requests.get("https://httpbin.org/robots.txt")

Expected Behavior

Code executed without error. urllib and requests parse system proxy as HTTP proxy.

Actual Behavior

urllib parsed system proxy as HTTPS proxy and throw an error.

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "C:\Python39\lib\site-packages\urllib3\connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Python39\lib\site-packages\urllib3\connection.py", line 496, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "C:\Python39\lib\site-packages\urllib3\util\ssl_.py", line 432, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Python39\lib\site-packages\urllib3\util\ssl_.py", line 474, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Python39\lib\site-packages\urllib3\util\retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /robots.txt (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /robots.txt (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1123)')))

There are several issues around like psf/requests#5740 and pypa/pip#9216 (comment) , from those issues I was told it's a urllib issue.

[sigmavirus24]

Can you confirm the output of

python -c 'import urllib; print(urllib.request.getproxies())'

Davy pointed at that being a step towards debugging but I haven't seen you provide that information

[kotori2]

Can you confirm the output of

python -c 'import urllib; print(urllib.request.getproxies())'

Davy pointed at that being a step towards debugging but I haven't seen you provide that information

it is in https. But since I didn't set any https_proxy related environment variables, it was loaded directly from system proxy.

>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890', 'https': 'https://127.0.0.1:7890', 'ftp': 'ftp://127.0.0.1:7890'}

[kotori2]

Oh so it's a urllib issue instead of urllib3 issue.

https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L2691

[kotori2]

For reference: https://bugs.python.org/issue42627

[ValdikSS]

This finally has been fixed in Python 3.10.5. All previous versions (in all branches) are still affected.
This should be workarounded in urllib3, as even pip uses this library and it fails to work right after the installation if proxy is configured.