connector.py file doesn't correctly set CONNECT method

[rookiemann]

Long story short

My problem is I really want to use aiohttp with asyncio to get Bing search engine results. My biggest problem is I've been trying to figure this out for days now. I've looked over many, MANY different discussions on the topic here (mostly here) and elsewhere.

Expected behaviour

The expected behavior is - using publicly available proxies (which can be many different type like Socks and regular CONNECT proxies etc. I do have a internal proxy server (http://127.0.0.1:8080) I'm using now to test, it gets public proxies, tests them and sets it on the server proxy. This server works fine.

I think for just a moment this morning I had it, I was testing against whatsmyip.org and I believe I did see I was 'broadcasting' from a different IP, but just like that I can't get it to work. I don't know what's going on. I am a novice, that is a problem as well too.

Actual behaviour

At first, I tried connecting using both http and https methods I was using for just regular 'request' method ... which was working out for me. When I wanted to switch to asyncio/aiohttp method I see I can't use https and need to use CONNECT method to tunnel in (I guess on :443?) using the ProxyConnector and setting up the proxy in the ClientSession.

Steps to reproduce

So, I'm using this guys code here which I like and it works great without proxies. I would like to use public proxies with this code here: https://fadeevab.com/18-lines-of-the-powerful-request-generator-with-python-asyncio-aiohttp/

import aiohttp
import asyncio

MAXREQ = 500
MAXTHREAD = 50
URL = 'https://google.com'

g_thread_limit = asyncio.Semaphore(MAXTHREAD)


async def worker(session):
    async with session.get(URL) as response:
        await response.read()


async def run(worker, *argv):
    async with g_thread_limit:
        await worker(*argv)


async def main():
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[run(worker, session) for _ in range(MAXREQ)])


if __name__ == '__main__':
    # Don't use asyncio.run() - it produces a lot of errors on exit.
    asyncio.get_event_loop().run_until_complete(main())

OK, so with this code above works fine, I need to connect public proxies to it.

Your environment

This here is the code I'm trying, it doesn't connect and I get back my own IP address each time. This is the code I was using that seems to have gotten me the closest to what I'm trying to accomplish ... but it's still telling me MY IP and not an IP from my proxy server. I have to put the code conn = None to get going, that seems sloppy to me though, not sure.

import aiohttp
import asyncio
import re
from bs4 import BeautifulSoup
from aiohttp_proxy import ProxyConnector, ProxyType

MAXREQ = 1
MAXTHREAD = 1
URL = 'https://www.whatismybrowser.com/detect/ip-address-location'

g_thread_limit = asyncio.Semaphore(MAXTHREAD)
conn = None

async def worker(session):
    conn == ProxyConnector("http://127.0.0.1:8080")

    async with session.get(URL) as response:
        
        await response.read()
        response = await response.read()
        soup = BeautifulSoup(response, "html.parser")
        print(soup)


async def run(worker, *argv):
    async with g_thread_limit:
        await worker(*argv)


async def main():
    async with aiohttp.ClientSession(connector=conn) as session:
        await asyncio.gather(*[run(worker, session) for _ in range(MAXREQ)])



if __name__ == '__main__':
    # Don't use asyncio.run() - it produces a lot of errors on exit.
    asyncio.get_event_loop().run_until_complete(main())

Sincerest apologies!

It took me days to reach out for help, I know I needed to really put some work in first before contacting, 3 days was enough frustration to bring me here. I hope I'm not irratating anyone with my noobness, I just think aiohttp is fantastic and want to use it.

Thanks for any help, it is really REALLY appreciated.

[webknjaz]

from aiohttp_proxy import ProxyConnector, ProxyType

You use a third-party proxy lib so you should probably ask them how to use it.

Also, this place is for bug reports. It's best to ask for help via Gitter/StackOverflow/Mailing lists.

[rookiemann]

Ah ok, my bad .. I will ask there. Thanks for your reply :)

[rookiemann]

Btw, I didn't mean to use the 3rd party stuff, I want to just use the original files, seems I can't connect to it?

Per th docs:
https://docs.aiohttp.org/en/v0.18.1/client_reference.html#aiohttp.ProxyConnector

I'm still trying to get a connection but I'm getting yelled at:

Traceback (most recent call last):
  File "c:/Users/Stryker/Desktop/py/scratch4a.py", line 57, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "C:\Users\Stryker\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 608, in run_until_complete     
    return future.result()
  File "c:/Users/Stryker/Desktop/py/scratch4a.py", line 50, in main
    conn = ProxyConnector(proxy="http://127.0.0.1:8080")
NameError: name 'ProxyConnector' is not defined

     conn = ProxyConnector(proxy="http://127.0.0.1:8080")
     async with aiohttp.ClientSession(connector=conn) as session:

[rookiemann]

Is this a bug? I don't understand why I can't

from aiohttp import ProxyConnector

[rookiemann]

I'm trying to figure out how to rectify this:

[rookiemann]

OK, I've been going in circles and I do think there is a bug now. From the very beginning I have been doing the correct way of connecting proxies to aiohttp. It is NOT the way to do the ProxyConnector anymore, I just have to add the proxy in to the session.get parameters like

proxy='http://127.0.0.1:8080'

I have been doing this from the very start but I was getting this error:
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host bing.com:443 ssl:False [The parameter is incorrect]

OK, this had me thinking I needed to do some extra steps so I looked and looked ... but in the wrong places and went off on tangents exploring 3rd party stuff etc.

I looked into aiohttps's files and I see there actually IS a CONNECT method in there to make the switch from http to https in the connector.py file .. but I think it's broken? Because it's not working, I cannot connect to sites that are https, but have no problem connecting to http site.

I think this is an issue?

[webknjaz]

You linked a 5-year-old version docs. Have you checked docs of the current stable aiohttp version? https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support

[rookiemann]

Yes sir I did, it was the first way I did it and I'm getting the error I last described. I unfortunately went hunting for and answer to why the current way wasn't working for me and stumbled upon old documents and getting further confused with how to remedy my problem.

I am having this problem doing it the new way:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host bing.com:443 ssl:False [The parameter is incorrect]