New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no response from phantomjs while task has fetched result #213

Open
xesina opened this Issue May 31, 2015 · 9 comments

Comments

Projects
None yet
3 participants
@xesina

xesina commented May 31, 2015

In some cases result has been fetched successfully but task has the folowing track :
"exception": "no response from phantomjs"

@binux

This comment has been minimized.

Owner

binux commented May 31, 2015

More detail please.
deployment / load / frequency of error / same project or not etc...

@xesina

This comment has been minimized.

xesina commented May 31, 2015

postgres
redis for queue
3 fetcher
3 processor
2 phantom
also using rotating proxy with tor/delegared/haproxy
load??
60%
yes for now i just have only one project
img

@binux

This comment has been minimized.

Owner

binux commented May 31, 2015

how many requests per minutes of phantomjs and the average crawl time of
each request. the error rate

sina saeedi notifications@github.com于2015年5月31日周日 下午5:02写道:

postgres
redis for queue
3 fetcher
3 processor
2 phantom
also using rotating proxy with tor/delegared/haproxy
load??
60%
yes for now i just have only one project
[image: img]
https://cloud.githubusercontent.com/assets/1429481/7901369/6963112c-0799-11e5-8b75-7ef091c95925.png


Reply to this email directly or view it on GitHub
#213 (comment).

@xesina

This comment has been minimized.

xesina commented May 31, 2015

1.3/3.0 rate/burst
30-40 request/minute to phantomjs
average crawl(fetch) time of each request : 15s
fetch_error rate: 70%

@binux

This comment has been minimized.

Owner

binux commented May 31, 2015

Did you try use more phantomjs instances to reduce the concurrent requests to phantomjs.
And did you find any error messages in phantomjs log?

Uncomment

console.debug = function(){};
may help to trace the requests made by phantomjs.

@binux

This comment has been minimized.

Owner

binux commented May 31, 2015

Uncomment -> Comment

@xesina

This comment has been minimized.

xesina commented May 31, 2015

Yes there is 3 phantomjs instances now, I have checked again, up to 80% of requests return FETCH_ERROR but results fetched successfully.
I'll try to debug phantomjs requests, thanks.

@potatoker

This comment has been minimized.

potatoker commented Nov 11, 2015

could you please tell me how I can open more phantomjs instances and make them work properly with the fetcher?

@binux

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment