Repository navigation
Using multiple proxies with arun_many() #626
Unanswered
Apapora
asked this question in
Forums - Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
In issue #246 there is this example to bypass 429 "too many requests" errors:
This code will crawl many urls using only one of the proxy addresses in PROXY_LIST, which quickly triggers 429 responses in my testing. Is there a way for the crawler to fully utilize all available proxies intelligently when using arun_many()? Say I have a list of 5 proxies and 'urls' is a list of 10 URLs, ideally the crawler would know to send 2 requests per each proxy. Or instead of using arun_many(), should I keep calling arun() with my separate rotating proxy logic? Thanks for any help with this.
All reactions