New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--autoreload not working since 3.1, change to non-blocking inotify needed #1658

Closed
AdrianLC opened this Issue Nov 17, 2013 · 9 comments

Comments

Projects
None yet
6 participants
@AdrianLC

AdrianLC commented Nov 17, 2013

The option is ignored because of the changes in autoreload.py from this in 3.0
to the current code.

According to Ask, if we want this option available again, a patch is needed for:

  • Changing the implementation, and use the non-blocking methods of pyinotify instead.
  • Add the watcher to the event loop.
@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Nov 18, 2013

Member

I tested and it works but the pool takes some time to catch up (each process in the pool must process one task before they are restarted).

The reload does not work if you change code in a task, but that is a limitation in the python reload function it seems.
That is, if I add a print statement to an existing task then that change is not applied, but if I add a completely
new then that new task will be available after reload.

Member

ask commented Nov 18, 2013

I tested and it works but the pool takes some time to catch up (each process in the pool must process one task before they are restarted).

The reload does not work if you change code in a task, but that is a limitation in the python reload function it seems.
That is, if I add a print statement to an existing task then that change is not applied, but if I add a completely
new then that new task will be available after reload.

@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Nov 18, 2013

Member

To fix the pool lag it should write one "noop" task to each process in the pool when a file changes, that way the processes will wake up from its waiting for I/O state.

Member

ask commented Nov 18, 2013

To fix the pool lag it should write one "noop" task to each process in the pool when a file changes, that way the processes will wake up from its waiting for I/O state.

@Thhhza

This comment has been minimized.

Show comment
Hide comment
@Thhhza

Thhhza Nov 18, 2013

Can confirm if the code in a task is changed, it takes one task to be processed after the change has been made before the change takes effect.

Thhhza commented Nov 18, 2013

Can confirm if the code in a task is changed, it takes one task to be processed after the change has been made before the change takes effect.

@alanhamlett

This comment has been minimized.

Show comment
Hide comment
@alanhamlett

alanhamlett Jun 2, 2014

Contributor

In my test on OS X with celery==3.1.11 and without pyinotify installed the --autoreload option does nothing.

This issue is related to #1025 and possibly #1880.

Contributor

alanhamlett commented Jun 2, 2014

In my test on OS X with celery==3.1.11 and without pyinotify installed the --autoreload option does nothing.

This issue is related to #1025 and possibly #1880.

@pashinin

This comment has been minimized.

Show comment
Hide comment
@pashinin

pashinin Sep 18, 2014

@ask can you help with as you said: "To fix the pool lag it should write one "noop" task to each process in the pool".
Where should I look? The only left bug I have with autoreloading is that a task is updated only after it's execution.

pashinin commented Sep 18, 2014

@ask can you help with as you said: "To fix the pool lag it should write one "noop" task to each process in the pool".
Where should I look? The only left bug I have with autoreloading is that a task is updated only after it's execution.

edmorley added a commit to mozilla/treeherder that referenced this issue May 10, 2016

Bug 1175757 - Remove unused pyinotify dependency from dev.txt
Since the celery auto-reload feature which it enables hasn't been
working for some time regardless:
celery/celery#1658
celery/celery#1025
celery/celery#1880

edmorley added a commit to mozilla/treeherder that referenced this issue May 11, 2016

Bug 1175757 - Remove unused pyinotify dependency from dev.txt
Since the celery auto-reload feature which it enables hasn't been
working for some time regardless:
celery/celery#1658
celery/celery#1025
celery/celery#1880
@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Jun 23, 2016

Member

Closing this, as we don't have the resources to complete this task. We will remove the docs for --autoreload and --autoscale in 4.0 (it will still be there, but no documented)

Member

ask commented Jun 23, 2016

Closing this, as we don't have the resources to complete this task. We will remove the docs for --autoreload and --autoscale in 4.0 (it will still be there, but no documented)

@ask ask closed this Jun 23, 2016

@alanhamlett

This comment has been minimized.

Show comment
Hide comment
@alanhamlett

alanhamlett Jun 26, 2016

Contributor

At least put a note in the docs saying --autoreload doesn't actually work?

Contributor

alanhamlett commented Jun 26, 2016

At least put a note in the docs saying --autoreload doesn't actually work?

@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Jun 27, 2016

Member

I put that in my todo list like a year ago. Currently I'm hard at work on Celery 4 which is nearly complete, and the option will be invisible, and undocumented in master very soon.

Member

ask commented Jun 27, 2016

I put that in my todo list like a year ago. Currently I'm hard at work on Celery 4 which is nearly complete, and the option will be invisible, and undocumented in master very soon.

@xeor

This comment has been minimized.

Show comment
Hide comment
@xeor

xeor Nov 1, 2017

bump :)

~ # celery worker --autoreload                     
usage: celery worker [options]                     
celery: error: unrecognized arguments: --autoreload

~ # celery --version                               
4.1.0 (latentcall)                                 

xeor commented Nov 1, 2017

bump :)

~ # celery worker --autoreload                     
usage: celery worker [options]                     
celery: error: unrecognized arguments: --autoreload

~ # celery --version                               
4.1.0 (latentcall)                                 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment