-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proxy support #176
Comments
Reqwest seems to support this. https://docs.rs/reqwest/latest/reqwest/struct.Proxy.html We could read the environment variable before creating the Downloader |
What I find strange, the environment variable is already read and processed somehow:
But, the downloaded page does not show the proxy ip for the ifconfig.me website. |
My knowledge on proxy is quite limited but I agree with you, something is not right |
From what I read here it should "just works"™ |
This works for me (with some warning and retry for the proxy connection):
The ip I get is not my real ip Note that I'm using https_proxy and not http_proxy. If I used your http_proxy with a random https_proxy this is not working. I think that for some reason we are doing https request even when specifying http. |
Nice catch, I can confirm it works with My public server IP got blocked from scraping a particular website, so I can tell it needs both kinds of proxies to circumvent the block. For future readers: |
Typical SuckIT Should we close this? |
As far as I am concerned, yes. |
I tried using a proxy to download via another IP, but couldn't get it to work via the
http_proxy
orHTTP_PROXY
environment variables:First, check "normal" public IP, then set a proxy, I tried some from this list until one worked: https://freeproxylists.net/
Check if the proxy works with the curl command (should return proxy IP).
Lastly, run suckit with the proxy and observe that the IP in the downloaded webpage is still the "normal" IP without the proxy.
Still, something is done with the proxy, since an invalid IP leads to an timeout or connection failure, and the latency is increased compared to a non-proxy run.
Beside this bug, a feature idea could be to offer multiple proxies to suckit, and the requests are split across the different proxies. This can further speed up downloading since less traffic is issued from a single IP.
The text was updated successfully, but these errors were encountered: