-
Notifications
You must be signed in to change notification settings - Fork 448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proxy connection is being refused #99
Comments
Just tested using scrapy to connect to the proxy instead of going through splash and that seems to work. |
@m2jobe I think the problem could be 127.0.0.1 address - Splash is usually executed in a Docker container, and localhost is different from localhost of the spider. |
See also: scrapinghub/splash#234 |
What if the proxy was executed in a docker container too? |
If a proxy is executed in a docker container too then you can use e.g. docker-compose to link these containers. You can have 'splash' and 'proxy' containers, link them, and proxy will be available at 'proxy:24000' address inside splash container. See e.g. how is it done in aquarium: https://github.com/TeamHG-Memex/aquarium/blob/master/%7B%7Bcookiecutter.folder_name%7D%7D/docker-compose.yml - it sets up Tor proxy and makes it available for all Splash containers under 'tor' hostname. |
Ok ill try that just tried the solution of using http://10.0.2.2, this led to a 504 error |
Following the linked yml example, is this correct?
`
|
Now this is the error Previously it was outputting "Proxy connection refused" |
Made some modifications, I think this is right: http://pastebin.com/pqiXGYN4 However still receiving the same "Error communicating with HTTP proxy" |
Finally got it working with this `version: '2' services:
|
Glad you got it working! Have you figured out what was the problem? |
Hi, everyone. |
This does not look specific to scrapy-splash. Shall we move this to https://github.com/scrapinghub/splash? |
You made my day man, been struggling with this as well for a few hours now and your solution works! |
@kmike If i'm using scraper api as a proxy do I need tor as a service running? What does tor do if you're running an external proxy? |
The error below suggest that my proxy connection is being refused. The proxy was tested with curl and it is infact working, it requires no credentials which is why the username and password fields were omitted in set_proxy. What else could be the reason for this connection being refused?
--ERROR
RenderErrorInfo(type='Network', code=99, text='Proxy connection refused',
"message": "Lua error: [string "..."]:10: network99", "type": "LUA_ERROR", "error": "network99"},
--TESTING PROXY
curl --proxy 127.0.0.1:24000 "http://lumtest.com/myip.json"
--SPIDER CODE
The text was updated successfully, but these errors were encountered: