no response from phantomjs while task has fetched result

[xesina]

In some cases result has been fetched successfully but task has the folowing track :
"exception": "no response from phantomjs"

[binux]

More detail please.
deployment / load / frequency of error / same project or not etc...

[xesina]

postgres
redis for queue
3 fetcher
3 processor
2 phantom
also using rotating proxy with tor/delegared/haproxy
load??
60%
yes for now i just have only one project

[binux]

how many requests per minutes of phantomjs and the average crawl time of
each request. the error rate

sina saeedi notifications@github.com于2015年5月31日周日 下午5:02写道：

postgres
redis for queue
3 fetcher
3 processor
2 phantom
also using rotating proxy with tor/delegared/haproxy
load??
60%
yes for now i just have only one project
[image: img]
https://cloud.githubusercontent.com/assets/1429481/7901369/6963112c-0799-11e5-8b75-7ef091c95925.png

—
Reply to this email directly or view it on GitHub
#213 (comment).

[xesina]

1.3/3.0 rate/burst
30-40 request/minute to phantomjs
average crawl(fetch) time of each request : 15s
fetch_error rate: 70%

[binux]

Did you try use more phantomjs instances to reduce the concurrent requests to phantomjs.
And did you find any error messages in phantomjs log?

Uncomment

pyspider/pyspider/fetcher/phantomjs_fetcher.js

Line 17 in 734cb3b

console.debug = function(){};

may help to trace the requests made by phantomjs.

[binux]

Uncomment -> Comment

[xesina]

Yes there is 3 phantomjs instances now, I have checked again, up to 80% of requests return FETCH_ERROR but results fetched successfully.
I'll try to debug phantomjs requests, thanks.

[potatoker]

could you please tell me how I can open more phantomjs instances and make them work properly with the fetcher?

[binux]

@potatoker http://docs.pyspider.org/en/latest/Deployment/