--autoreload not working since 3.1, change to non-blocking inotify needed

[AdrianLC]

The option is ignored because of the changes in autoreload.py from this in 3.0
to the current code.

According to Ask, if we want this option available again, a patch is needed for:

• Changing the implementation, and use the non-blocking methods of pyinotify instead.
• Add the watcher to the event loop.

[ask]

I tested and it works but the pool takes some time to catch up (each process in the pool must process one task before they are restarted).

The reload does not work if you change code in a task, but that is a limitation in the python reload function it seems.
That is, if I add a print statement to an existing task then that change is not applied, but if I add a completely
new then that new task will be available after reload.

[ask]

To fix the pool lag it should write one "noop" task to each process in the pool when a file changes, that way the processes will wake up from its waiting for I/O state.

[Thhhza]

Can confirm if the code in a task is changed, it takes one task to be processed after the change has been made before the change takes effect.

[alanhamlett]

In my test on OS X with celery==3.1.11 and without pyinotify installed the --autoreload option does nothing.

This issue is related to #1025 and possibly #1880.

[pashinin]

@ask can you help with as you said: "To fix the pool lag it should write one "noop" task to each process in the pool".
Where should I look? The only left bug I have with autoreloading is that a task is updated only after it's execution.

[ask]

Closing this, as we don't have the resources to complete this task. We will remove the docs for --autoreload and --autoscale in 4.0 (it will still be there, but no documented)

[alanhamlett]

At least put a note in the docs saying --autoreload doesn't actually work?

[ask]

I put that in my todo list like a year ago. Currently I'm hard at work on Celery 4 which is nearly complete, and the option will be invisible, and undocumented in master very soon.

[xeor]

bump :)

~ # celery worker --autoreload                     
usage: celery worker [options]                     
celery: error: unrecognized arguments: --autoreload

~ # celery --version                               
4.1.0 (latentcall)