Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

some questions with proxy when i use scrapy-splash #181

Closed
AceXia opened this issue Jul 10, 2018 · 3 comments
Closed

some questions with proxy when i use scrapy-splash #181

AceXia opened this issue Jul 10, 2018 · 3 comments

Comments

@AceXia
Copy link

AceXia commented Jul 10, 2018

i want to use scrapy-splash with proxy,my code like this:
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse,args={'wait': 10,'proxy': 'http://127.0.0.1:1080'})

but i get some messsage in console like this: 502 Bad Gateway
and splash show some message :
2018-07-10 16:00:16.670873 [events] {"qsize": 0, "load": [0.02, 0.04, 0.0], "rendertime": 0.02126336097717285, "error": {"type": "RenderError", "info": {"text": "Proxy connection refused", "code": 99, "url": "https://www.baidu.com", "type": "Network"}, "description": "Error rendering page", "error": 502}, "path": "/render.html", "_id": 139738842022520, "fds": 20, "args": {"proxy": "http://127.0.0.1:1081", "wait": 10, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8", "User-Agent": "Scrapy/1.5.0 (+https://scrapy.org)", "Accept-Language": "en"}, "uid": 139738842022520, "url": "https://www.baidu.com"}, "maxrss": 149472, "user-agent": "Scrapy/1.5.0 (+https://scrapy.org)", "method": "POST", "client_ip": "172.17.0.1", "timestamp": 1531238416, "status_code": 502, "active": 0}
2018-07-10 16:00:16.672596 [-] "172.17.0.1" - - [10/Jul/2018:16:00:15 +0000] "POST /render.html HTTP/1.1" 502 269 "-" "Scrapy/1.5.0 (+https://scrapy.org)"

who can help me ,please

@Rockyzsu
Copy link

i met such kind of issue, and found it was docker splash issue.

@xavier90
Copy link

I think it might because the bad ip proxy, try to change to another one.

@Gallaecio
Copy link
Contributor

@AceXia Did you ever find a solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants