Celery integration

[amishHammer]

As per the discussion on background tasks in issue #1097 this provides the base framework for integrating Celery into InvenTree along with some invoke targets to make it easy for someone to get it up and running.

[SchrodingersGat]

Tom I like what you have started here and I am very keen to see a celery integration for the next release. There are a few other projects in the works that would really benefit from this!

A few points of contention ;)

Installing

If we are to include the celery requirement, it has to be reasonably easy for the user to install.

I see you have created installation functions in tasks.py (which is great!).

These look like will only work under a system which has apt-get support - we'd need to consider how other operating systems could be supported.

Running

How do we ensure that the celery worker server is running whenever the InvenTree server is launched?

This is (currently) my major concern, that we are imposing a system which is more difficult to get up and running, and to troubleshoot in the case that something goes wrong.

Ideally, I would like the InvenTree server to check (on startup) if the celery worker is running, and start it, if it is not running. Exactly how it starts would probably depend on what worker the user wants to use...

Is something like this possible?

[amishHammer]

Tom I like what you have started here and I am very keen to see a celery integration for the next release. There are a few other projects in the works that would really benefit from this!

A few points of contention ;)

Discussion is health!

Installing

If we are to include the celery requirement, it has to be reasonably easy for the user to install.

Agreed, installation needs to be as seamless as possible.

I see you have created installation functions in tasks.py (which is great!).

These look like will only work under a system which has apt-get support - we'd need to consider how other operating systems could be supported.

Currently all existing functions in tasks.py only support apt-get based distributions, so this was what was replicated for the components for Celery support.

IMHO installation support for a variety of OS's is a separate issue to this. The invoke interface is cute and works well for some tasks but for installing dependencies it is not the best solution. There are plenty of ways to skin this issue below are a few solutions most of which should be considered out of scope for this PR/project:

• Provide 'better' installation instructions in the docs project with distro specific instructions on getting things going
• Provide a manifest for a systems management tool (puppet/ansible/cf-engine/etc/etc) that pre-configures the system for development work
• Provide re-invent the basics of the systems management tool in tasks.py for platform detection

Personally I think installation type tasks should be removed from tasks.py and should be moved into more fully featured installation instructions.

Running

How do we ensure that the celery worker server is running whenever the InvenTree server is launched?

I can add a error/warning box to the web interface to check on the status of the Celery worker(s) and display an error to the user when the workers are not running. This should also help with the concern below in notifying the user that there is an issue with the back end.

This is (currently) my major concern, that we are imposing a system which is more difficult to get up and running, and to troubleshoot in the case that something goes wrong.

Ideally, I would like the InvenTree server to check (on startup) if the celery worker is running, and start it, if it is not running. Exactly how it starts would probably depend on what worker the user wants to use...

Is something like this possible?

For development mode we could:

• Patch manage.py to spawn a worker when running the server
• Patch tasks.py to spawn the worker when running the server
• Rely on the existing worker task in tasks.py to run the server and warn the user that it is not running as above.

Personally I prefer the last option for 2 reasons:

• Managing the sub processes from invoke/manage is more complex than ideal, you have to leave a master process in control to accept the signal and pass it along to both workers. There would need to be some form of crash detection to detect one of the 2 processes restarting. We would need to implement a filesystem watchdog with something like watchmedo to monitor the filesystem and restart the Celery worker when code changes so that it acts the same as the django instance.
• Running both processes on a single console will cause the logging of the webserver and the worker to get interleaved making it more difficult to discern what is taking place and which process the code is running in. This can somewhat be elevated with some more advanced logging configuration but it is still more painful than having 2 processes in separate terminals.

In order to make it easier for people we could provide an example tmux configuration for people that spawns both process for the system in separate tabs/split screens.

For production deployments we will need to provide an init/systemd/supervisord script/configuration to start the worker as neither tasks.py or manage.py will be used to run the django application under a production web server such as apache or nginx.

[SchrodingersGat]

Currently all existing functions in tasks.py only support apt-get based distributions, so this was what was replicated for the components for Celery support.

Fair enough!

Personally I think installation type tasks should be removed from tasks.py and should be moved into more fully featured installation instructions.

I think this is a sensible solution. There are going to be numerous subtle differences between different installations and we can push the responsibility off to the person installing to get it right for their installation...

I can add a error/warning box to the web interface to check on the status of the Celery worker(s) and display an error to the user when the workers are not running. This should also help with the concern below in notifying the user that there is an issue with the back end.

I really like this idea.

Something that is front-and-center on every page saying that "hey, something's wrong!"

• Should appear for all users (even if they don't have admin rights to change it) so they can at least tell the sysadmin
• A generic implementation of a "warnings exist" function so that in future we can add other process checks
• In the "Statistics" tab (available under the user menu) add a status indicator for each function we want a status report on (e.g. celery workers)
• Rename "Statistics" to something more appropriate like "System Info"

Having a warning which clearly says that "your celery worker isn't running" (and a link to clear documentation which shows how to get it running) is the cleanest solution here I think...

[SchrodingersGat]

@amishHammer I have implemented a simple framework for displaying "health status" of the server. If the InvenTree server cannot communicate with a celery worker thread, the user can now be alerted.

#1136

[rcludwick]

This is (currently) my major concern, that we are imposing a system which is more difficult to get up and running, and to troubleshoot in the case that something goes wrong.

Ideally, I would like the InvenTree server to check (on startup) if the celery worker is running, and start it, if it is not running. Exactly how it starts would probably depend on what worker the user wants to use...

So if something isn't capable of running in a docker container, I'm probably not going to run it. I imagine that's the case for others as well. I would suggest that as being a way to allow people to easily install it without having to install and configure out a rabbitmq server by hand, keeping packages up to date, etc. etc.

I recently did a clean install of ubuntu, moved the docker compose files and data config files over, ran docker-config up -d and it all 10 or so docker servers came up without issue. It was the quickest server migration I ever did.

Rabbitmq needs to run in it's own container. Supervisord should be used for guaranteeing the celery workers (and gunicorn for that matter) are running in the container.

Then the container just starts supervisord with a config, and let supervisord figure out whether things are running or not.

A health check for the celery workers is fine but you probably shouldn't try to re-invent the wheel for restarting the celery workers especially since something robust enough for production use is out there.

http://supervisord.org/