Failed to mirror package via Proxy Server since version 4.0.0

[alebourdoulous]

Since version 4.0.0 of bandersnatch , I can't use my proxy with environment variables
http_proxy or https_proxy

root@01e29aacbf76:/# bandersnatch --debug mirror
2020-05-04 11:36:23,022 DEBUG: Initializing Master's aiohttp ClientSession
2020-05-04 11:36:23,023 INFO: Status file /srv/pypi/status missing. Starting over.
2020-05-04 11:36:23,023 INFO: Syncing with https://pypi.org.
2020-05-04 11:36:23,023 INFO: Current mirror serial: 0
2020-05-04 11:36:23,024 INFO: Syncing all packages.
2020-05-04 11:36:23,031 DEBUG: Closing Master's aiohttp ClientSession
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 494, in sock_connect
    return await fut
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 526, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 443)
.....

[fredbcode]

Works well with 3.6.0, introduced in version 4.0.0

[cooperlees]

Hi, thanks for reporting. 4.0.0 got a complete move from requests (urllib3) to aiohttp. So, here it seems you use a HTTPS proxy? It seems so from the connect to local host on 443:

raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 443)

Can you give me a little more detail:

• what exactly are you setting the variables to?
• what proxy server are you using?

I’m going to check two things here:

• If proxies are being set correctly with the XMLRPC library that sits on top of aiohttp (according to the log that’s where you’re failing)
• Check if aiohttp can support https proxies ... I do remember bugs there

If you are using https can you try with http or socks protocols?

[alebourdoulous]

Hi Cooper,
Thank you for your quick response, we can't join the proxy, no packet is sent to our Bluecoat proxy.
https_proxy=http://100.100.100.100:8000
http_proxy=http://100.100.100.100:8000

Best Regard's

[cooperlees]

Seen we do not respect the environment vars for proxy settings for the xmlrpc calls. I've put up a quick workaround, but I feel the real fix is to use self.session from Master rather than create a separate aiohttp client session for aiohttp-xmlrpc.

To test my workaround you can via:

• pip install git+https://github.com/pypa/bandersnatch.git@xmlrpc_proxy

Would love to know if that makes it work.

[alebourdoulous]

Good news it works very well.

root@3f3ae77c22b3:/# bandersnatch --version
bandersnatch 4.0.2

root@3f3ae77c22b3:/# bandersnatch mirror
2020-05-04 20:54:26,431 INFO: Setting up mirror directory: /srv/pypi
2020-05-04 20:54:26,432 INFO: Setting up mirror directory: /srv/pypi/web/simple
2020-05-04 20:54:26,432 INFO: Setting up mirror directory: /srv/pypi/web/packages
2020-05-04 20:54:26,432 INFO: Setting up mirror directory: /srv/pypi/web/local-stats/days
2020-05-04 20:54:26,433 INFO: Generation file missing. Reinitialising status files.
2020-05-04 20:54:26,434 INFO: Status file /srv/pypi/status missing. Starting over.
2020-05-04 20:54:26,434 INFO: Syncing with https://pypi.org.
2020-05-04 20:54:26,434 INFO: Current mirror serial: 0
2020-05-04 20:54:26,434 INFO: Syncing all packages.
2020-05-04 20:54:36,127 INFO: No project filters are enabled. Skipping filtering
2020-05-04 20:54:36,128 INFO: Trying to reach serial: 7167657
2020-05-04 20:54:36,128 INFO: 232068 packages to sync.
...

Thank you so much

[alebourdoulous]

I suppose the xmlrpc_proxy branch well be merge in master soon ?

Thank's
Alain

[cooperlees]

I'll try get it in there and release this week. I'm going to fix the aiohttp-xmlrpc client to use the same client session as the main file downloading to only have one aiohttp session open during the mirror run.

[redbaron4]

I have a similar issue because we use a socks5 proxy in our environment. With the latest 4.1.0, I get the error ValueError: Only http proxies are supported which is raised by aiohttp.

The command we use to run bandersnatch (worked OK in 3.6)

sudo -u apache https_proxy=socks5h://192.168.2.11:1080 /opt/bandersnatchenv2/bin/bandersnatch mirror

ANy insights will be appreciated.

[cooperlees]

In 4.0 we went from the requests library to aiohttp. As stated, it seems aiohttp does not support socks proxy servers (sorry).

Can you enable http on your proxy? If not, I’ll accept PRs adding socks support to aiohttp if we can.

[redbaron4]

It is not possible to enable http on the proxy so I guess I'll stick to 3.6 for the time being.

There are however socks connectors for aiohttp, like this . Would it be possible to use a connector with existing code?

[cooperlees]

Sure would be possible. I'll open an issue.