Replies: 2 comments
-
I have also tried:
Before each request to no avail |
Beta Was this translation helpful? Give feedback.
0 replies
-
I would suggest using multiple clients for this use case. You could also handle this functionality using a single client with a custom transport class that rotates outgoing requests across multiple sub-transports, each configured to a different proxy port. (Perhaps there's scope for a third party package or example GitHub gist there.) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a proxy provider that has provided me x number of proxies, authenticated as follows:
http://uname:password@hostname:port[10000-10014]
I have no problem setting this up as follows:
proxies = {'http://': 'http://uname:password@hostname:port', 'https://': 'http://uname:password@hostname:port'}
However, my provider limits me on the number of threads on a PORT basis, so I wrote a function called get_proxy_address that selects a random port in a valid range (for my specific case, they are ports 10000 to 10014, giving me 15 possible ports to use)
I am using httpx in an async fashion, like this:
async with httpx.AsyncClient(proxies=get_proxy_address(), verify=False) as session:
However, the get_proxy_address() function returns just one of the proxies, since the function is only evaluated once, and only one of them gets used for the entire async session.
I removing proxies= from the instantiation of http.AsyncClient as session and tried adding it to session.get(url, proxies=proxies), but that gives me the following error:
Any ideas on how I could rotate through my 15 ports in a singular httpx.AsyncClient session?
Thank you so much in advance!
Beta Was this translation helpful? Give feedback.
All reactions