The requests library is not working while the global proxy is enabled

[AntiOs314]

Problem Description

Once the Windows "Use a proxy server" option is enabled,

requests library in my addon does not work.
Example1:

import requests

class example:
  def __init__(self):
    print('example loaded!')

    r = requests.get('https://mitmproxy.org/', timeout=5)
    print(r)

    print('Done!')

addons = [
  example()
]

example loaded!
Loading script .\solve.py
in script .\solve.py: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002ACDF46B040>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it')))
Proxy server listening at http://*:8080

However, when I disable the "Use a proxy server" option(which means traffic will not pass through mitmproxy),

example loaded!
<Response [200]>
Done!

Example2:
If I want to send a request every time mitmproxy catch a request to 'github.com'("Use a proxy server" is enabled)：

import requests

class example:
  def __init__(self):
    print('example loaded!')
  def request(self, flow):
    if flow.request.host == 'github.com':
      r = requests.get('https://mitmproxy.org/', timeout=5)
      print('github!')


addons = [
  example()
]

Addon error: Traceback (most recent call last):
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connection.py", line 364, in connect
    self.sock = conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connection.py", line 499, in _connect_tls_proxy
    socket = ssl_wrap_socket(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:1112: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1112: The handshake operation timed out')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".\solve.py", line 8, in request
    r = requests.get('https://mitmproxy.org/', timeout=5)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\adapters.py", line 513, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1112: The handshake operation timed out')))

Example3:

import requests

class example:
  def __init__(self):
    print('example loaded!')

  def response(self, flow):
    print('response!')
    if flow.request.host == 'github.com':
      r = requests.get('https://mitmproxy.org/')
      print(r)
      print('github!')


addons = [
  example()
]

And this blocks mitmproxy and my browser.
However, if I add timeout=5 to r = requests.get('https://mitmproxy.org/'), this example turns into example 2 above.

Similar issue on github:
#4572
Similar issues on stackoverflow:
https://stackoverflow.com/questions/66089365/requests-library-does-not-perform-http-requests-inside-mitmproxy
https://stackoverflow.com/questions/55066451/mtmproxy-http-get-request-in-script
https://stackoverflow.com/questions/66089365/requests-library-does-not-perform-http-requests-inside-mitmproxy

System Information

Mitmproxy: 8.0.0
Python: 3.9.7
OpenSSL: OpenSSL 1.1.1n 15 Mar 2022
Platform: Windows-10-10.0.19044-SP0

[KORraNpl]

I believe requests uses its own bundled certificate store, not the one from OS (EDIT: apparently not anymore, but my comment is still valid). So even if you added mitmproxy CA in your system, it's not trusted by requests. If you search for "python requests self-signed certificate", there are a couple of solutions. I usually run my scripts this way:

SSL_CERT_FILE=<path_to_mitmproxy_ca> https_proxy=https://127.0.0.1:8080 python3 my_script.py

[Prinzhorn]

You are blocking the response hook with your synchronous requests.get call, effectively locking up mitmproxy. All hooks are processed synchronously. This is only a problem because requests.get itself goes through the proxy, which is blocked by its own call. You can use async hooks to prevent that, e.g. async def response(self, flow):. The (deleted) answer in the SO posts essentially does that but with the @concurrent decorator. But the new core uses asyncio, so I recommend async hooks.

[mhils]

FWIW, httpx is a great alternative to requests which I would recommend checking out. :)