Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

requests library not working while global proxy enabled #5330

Closed
AntiOs314 opened this issue May 7, 2022 · 1 comment
Closed

requests library not working while global proxy enabled #5330

AntiOs314 opened this issue May 7, 2022 · 1 comment
Labels
kind/triage Unclassified issues

Comments

@AntiOs314
Copy link

Problem Description

Once the Windows "Use a proxy server" option is enabled,
image
requests library in my addon will not work.
Example1:

import requests

class example:
  def __init__(self):
    print('example loaded!')

    r = requests.get('https://mitmproxy.org/', timeout=5)
    print(r)

    print('Done!')

addons = [
  example()
]
example loaded!
Loading script .\solve.py
in script .\solve.py: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002ACDF46B040>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it')))
Proxy server listening at http://*:8080

However, when I disable the "Use a proxy server" option(which means traffic will not pass through mitmproxy),

example loaded!
<Response [200]>
Done!

Example2:
If I want to send a request every time mitmproxy catch a request to 'github.com'("Use a proxy server" is enabled):

import requests

class example:
  def __init__(self):
    print('example loaded!')
  def request(self, flow):
    if flow.request.host == 'github.com':
      r = requests.get('https://mitmproxy.org/', timeout=5)
      print('github!')


addons = [
  example()
]
Addon error: Traceback (most recent call last):
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connection.py", line 364, in connect
    self.sock = conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connection.py", line 499, in _connect_tls_proxy
    socket = ssl_wrap_socket(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Users\11316\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:1112: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1112: The handshake operation timed out')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".\solve.py", line 8, in request
    r = requests.get('https://mitmproxy.org/', timeout=5)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\11316\.local\pipx\venvs\mitmproxy\lib\site-packages\requests\adapters.py", line 513, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='mitmproxy.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1112: The handshake operation timed out')))

Example3:

import requests

class example:
  def __init__(self):
    print('example loaded!')

  def response(self, flow):
    print('response!')
    if flow.request.host == 'github.com':
      r = requests.get('https://mitmproxy.org/')
      print(r)
      print('github!')


addons = [
  example()
]

And this blocks mitmproxy and my browser.
However, if I add timeout=5 to r = requests.get('https://mitmproxy.org/'), this example turns into example 2 above.

Similar issue on github:
#4572
Similar issues on stackoverflow:
https://stackoverflow.com/questions/66089365/requests-library-does-not-perform-http-requests-inside-mitmproxy
https://stackoverflow.com/questions/55066451/mtmproxy-http-get-request-in-script
https://stackoverflow.com/questions/66089365/requests-library-does-not-perform-http-requests-inside-mitmproxy

System Information

Mitmproxy: 8.0.0
Python: 3.9.7
OpenSSL: OpenSSL 1.1.1n 15 Mar 2022
Platform: Windows-10-10.0.19044-SP0

@AntiOs314 AntiOs314 added the kind/triage Unclassified issues label May 7, 2022
@KORraNpl
Copy link
Contributor

KORraNpl commented May 7, 2022

I believe requests uses its own bundled certificate store, not the one from OS (EDIT: apparently not anymore, but my comment is still valid). So even if you added mitmproxy CA in your system, it's not trusted by requests. If you search for "python requests self-signed certificate", there are a couple of solutions. I usually run my scripts this way:

SSL_CERT_FILE=<path_to_mitmproxy_ca> https_proxy=https://127.0.0.1:8080 python3 my_script.py

@mitmproxy mitmproxy locked and limited conversation to collaborators May 7, 2022
@mhils mhils converted this issue into discussion #5331 May 7, 2022

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
kind/triage Unclassified issues
Projects
None yet
Development

No branches or pull requests

2 participants