How to use HTTPS proxy with aiohttp?

[wangybnet]

I noticed the code in ProxyConnector：
assert proxy.startswith('http://'), ("Only http proxy supported", proxy)
Is there any way to use HTTPS proxy with aiohttp?
If not, is this feature under consideration?
Any help will be appreciated.

[popravich]

Currently there is no way to use HTTPS proxies.

[wangybnet]

Thanks anyway :)

[mikenerone]

Could someone shed some light on why HTTPS proxies are unsupported? I might be able to help out with a PR, but I want to understand whatever issue underlay this decision.

[asvetlov]

Perhaps the main problem is that proxies are covered by mocked tests only.
aiohttp test suite just doesn't detect proxy regressions.
I see the only option: test via real proxy.
Tests should be skipped if proxy is not configured but travis should be configured to use proxies at least.
Aslo testing page should have instructions for proxy server setup.

If you want work on it -- it would be awesome!

[fafhrd91]

We can write simple proxy, so we can use it for tests and as an example

[asvetlov]

@fafhrd91 it is the best solution but who will do it?
I could be a volunteer if KeepSafe will pay for the task. Right now I'm overwhelmed by other aiohhtp issues and they are more important than proxy support, isn't it?

[fafhrd91]

I think this task is low priority in general, also KeepSafe does not use any proxies. I am not with KeepSafe anymore, so I can not comment on payment.

@danielnelson is responsible for KeepSafe backend

[mgwilliams]

I think #1366 may not fix this, now that I read over it again. #1366 fixes use of CONNECT tunnelling of an https server through an http proxy (recently broken).

[mattharrison]

#1366 did not work for me. I get a ProxyConnectionError(1, 'Can not connect to localhost:8081 [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)]') error. I'm assuming that I need to point to the cert somehow. In requests I set a REQUESTS_CA_BUNDLE environment variable.

[fafhrd91]

try verify_ssl=False

[mattharrison]

@fafhrd91 thanks for the suggestion. I need to use a ProxyConnector object which doesn't expose access to the verify_ssl attribute of the BaseConnector. I tried setting the _verify_ssl attribute of ProxyConnector to False, but am still seeing the same issue. Will dig in a little more.

[fafhrd91]

ProxyConnector is deprecated and removed in master

[mattharrison]

Thanks for your response @fafhrd91 . Trying again with latest zeep and aiohttp code. Here's my relevant aiohttp code:

        if async:
            loop = asyncio.get_event_loop()
            auth = aiohttp.BasicAuth(username, password)
            if proxy: 
                conn = aiohttp.TCPConnector(verify_ssl=False)
                session = aiohttp.ClientSession(loop=loop, connector=conn)
                t = ProxyAsyncTransport(loop, session=session,
                                   proxy=url, auth=auth)
            else:
                session = aiohttp.ClientSession(loop=loop, auth=auth)
                t = AsyncTransport(loop, cache=None, session=session)

The ProxyAsyncTransport class looks like this:

class ProxyAsyncTransport(AsyncTransport):
    def __init__(self, loop, proxy, session=None, auth=None):
        super().__init__(loop, session=session)
        self.auth = auth
        self.proxy = proxy

    async def post(self, address, message, headers):
        self.logger.debug("HTTP Post to %s:\n%s", address, message)
        with aiohttp.Timeout(self.operation_timeout):
            if self.auth:
                response = await self.session.post(
                    address, data=message, headers=headers,
                    proxy=self.proxy, proxy_auth=self.auth)
            else:
                response = await self.session.post(
                    address, data=message, headers=headers,
                    proxy=self.proxy)
            self.logger.debug(
                "HTTP Response from %s (status: %d):\n%s",
                address, response.status, await response.read())
            return response

    async def get(self, address, params, headers):
        with aiohttp.Timeout(self.operation_timeout):
            if self.auth:
                response = await self.session.get(
                    address, params=params, headers=headers,
                    proxy=self.proxy, proxy_auth=self.auth)
            else:
                response = await self.session.get(
                    address, params=params, headers=headers,
                    proxy=self.proxy)
            from pretend import stub
            return await stub(
                content=await response.read(),
                status_code=response.status,
                headers=response.headers)

Am running into this Error:

Only http proxies are supported

If I comment out the line throwing that error in aiohhtp/client_regrep.py, I get:

[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:749)]

[mattharrison]

Looks like I have it working. I needed to change my proxy url to be http:// instead of https://. Also, for authentication, I needed to pass the auth parameter to ClientSession not to the session.get.

My working code looks like this:

        if async:
            loop = asyncio.get_event_loop()
            auth = aiohttp.BasicAuth(username, password)
            if proxy:
                url = url.replace('https://', 'http://')
                conn = aiohttp.TCPConnector(verify_ssl=False)
                #ctx = ssl.create_default_context(cafile=config['PROXY_CA_BUNDLE'])
                #conn = aiohttp.TCPConnector(ssl_context=ctx)
                session = aiohttp.ClientSession(loop=loop, connector=conn, auth=auth)
                t = ProxyAsyncTransport(loop, session=session,
                                   proxy=url)
            else:
                session = aiohttp.ClientSession(loop=loop, auth=auth)
                t = AsyncTransport(loop, cache=None, session=session)

[mikenerone]

@mattharrison If I'm understanding correctly, your example code doesn't address this use case. This issue is about proxying over an HTTPS proxy connection. I.e. the proxy URL is itself an https - the connection between the client and the proxy is itself TLS/SSL-encrypted. Squid is an example of a proxy server with this feature. Changing the proxy URL so it becomes proxy-over-http isn't the same thing at all.

All that said, IMO, proxying over HTTPS is ill-advised in almost all cases, and actually decreases security. When people want to use it, it is almost always a result of misunderstanding (not being aware of how CONNECT works to maintain security when proxying https over http proxy). Unfortunately, misunderstanding or not, there are business environments where proxy-over-https is required by policy.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.