New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

celery SoftTimeLimitExceeded support #84

Merged

ResolveWang merged 1 commit into SpiderClub:master from thekingofcity:master

Apr 1, 2018

Member

thekingofcity commented Apr 1, 2018 •

edited

This simply fix stuck tasks.

By adding '--soft-time-limit secs' when you start celery, this will resend task to broker.

e.g. celery -A tasks.workers -Q user_crawler worker -l info -c 1 --soft-time-limit 10

Now it only works on user_crawler task simply because crawling user detail costs O(1) in time.


          celery SoftTimeLimitExceeded support

644858f

ResolveWang merged commit 444eb27 into SpiderClub:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment