Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Beats worker not properly writing state on SIG TERM #1760

Closed
infecto opened this Issue · 4 comments

2 participants

@infecto

Spoke about this a bit via IRC with the result. commit 0c0daa5. Still have issues on our Heroku deployment. The worker state is not being correctly recorded to database scheduler (django celery). What kind of information can I provide to further this issue.

@ask
Owner

Are you sending TERM to the process?

@infecto

Unfortunately I am not 100% certain if Heroku is sending TERM correctly for every deployment. I assume not because the worker is not syncing state before deployment. Let me know how I can provide more information/feedback. I will try to dump some logs on a reboot to see whats going on.

@infecto
Feb 18 11:32:13 heroku app/worker1.1:  worker: Warm shutdown (MainProcess) 
Feb 18 11:32:13 heroku app/worker1.1:  [2014-02-18 11:32:13,360: INFO/Beat] Writing entries... 
Feb 18 11:32:13 heroku app/worker1.1:  [2014-02-18 11:32:13,389: INFO/MainProcess] beat: Shutting down... 
Feb 18 11:32:28 heroku app/worker1.1:   -------------- celery@domain.com v3.1.7 (Cipater) 
Feb 18 11:32:28 heroku app/worker1.1:  ---- **** -----  
Feb 18 11:32:28 heroku app/worker1.1:  --- * ***  * -- Linux-3.8.11-ec2-x86_64-with-debian-squeeze-sid 
Feb 18 11:32:28 heroku app/worker1.1:  -- * - **** ---  
Feb 18 11:32:28 heroku app/worker1.1:  - ** ---------- [config] 
Feb 18 11:32:28 heroku app/worker1.1:  - ** ---------- .> app:         default:0x22a4490 (djcelery.loaders.DjangoLoader) 
Feb 18 11:32:28 heroku app/worker1.1:  - ** ---------- .> transport:   redisurl 
Feb 18 11:32:28 heroku app/worker1.1:  - ** ---------- .> results:     database 
Feb 18 11:32:28 heroku app/worker1.1:  - *** --- * --- .> concurrency: 4 (prefork) 
Feb 18 11:32:28 heroku app/worker1.1:  -- ******* ----  
Feb 18 11:32:28 heroku app/worker1.1:  --- ***** ----- [queues] 
Feb 18 11:32:28 heroku app/worker1.1:   -------------- .> celery           exchange=celery(direct) key=celery 

@ask
Owner

You can now use CELERYBEAT_SYNC_EVERY to sync more frequently, and one bug with TERM was also fixed recently.

@ask ask closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.