Background jobs

[rtubio]

Although this is not initially one of the main tasks for a web framework, being able to run independent background jobs / tasks is a mechanism that provides a lot of flexibility when creating a web application.

• Are there any plans to integrate a job / task manager with Rocket? Anything similar to Celery+Django?
• As of today, which would be the recommended way to run these background tasks?

[SergioBenitez]

In general, there should rarely be a need for a specialized job/queue system with async Rust. The runtime, and async/await in general, make common "scheduling" tasks fairly straightforward. To start a background task, just spawn a task with tokio:

tokio::spawn(async {
   ...
});

To do something periodically, spawn a task with a loop that waits for a timer:

tokio::spawn(async {
    let mut interval = time::interval(Duration::from_secs(60 * 30));
    loop {
        interval.tick().await;
        // ...
    }
});

If you need more control, use the various synchronization primitives in tokio::sync like the managed queue example.

[rtubio]

Thanks for the examples!

As far as I know, tokio can run jobs / tasks within the same computing instance. My tasks are quite large and I would like to run them in a separate computing instance. At the same time, in my case, it is highly desirable to keep the state of execution of that task within rocket itself; as well as the results of the execution of the task.

• Can tokio achieve some similar degree of functionality?

This is something that I could accomplish with Django/Celery/Redis at a relatively low configuration cost. This combination allows me to run very large tasks in separated instances.

I have seen that Celery Rust is making some advances in terms of feature set.

• Are there any plans to integrate Rocket with Celery Rust or some other similar library, to be able to spawn those background jobs in a separate computing instance?

Thanks for the openness and for your time!

[SergioBenitez]

My tasks are quite large and I would like to run them in a separate computing instance. At the same time, in my case, it is highly desirable to keep the state of execution of that task within rocket itself; as well as the results of the execution of the task.

These two requirements seem to be at odds with each-other. To execute a task remotely, you need to transfer the necessary state to the remote, which performs the computation with a result local to the remote. To get that result back to the first machine, it needs to be transferred back. Anything else is an abstraction on top of this, including celery.

Can tokio achieve some similar degree of functionality?

Tokio is a generic async runtime and async primitives library. You can implement your own celery-like system on top of this, and doing so would be relatively straightforward for some core functionality. You could even just run a Rocket server on the remote. All of these things are effectively RPC servers. The challenge comes when you want an extensive feature-set, like redundancy, load balancing, worker pools, etc. But by leveraging tools like redis, implementing even these features becomes relatively straightforward. Of course, something like celery is a single package that makes doing this much easier and likely more robust with edge-case handling, etc.

Are there any plans to integrate Rocket with Celery Rust or some other similar library, to be able to spawn those background jobs in a separate computing instance?

Rocket is designed to make these kinds of extensions feel first class even when they are implemented externally to Rocket. For example, rocket_dyn_templates is external to Rocket yet feels more first class than similar functionality in many frameworks: it integrates with configuration, provides launch-time guarantees, requires no boilerplate beyond the absolute necessary, and otherwise "just works". You, or the celery library, could do the same for celery. It could be as straightforward as doing nothing but adding the celery "app" instance to managed state or as complete as the support in rocket_dyn_templates.

[rtubio]

Many thanks for the extensive and complete answer and please excuse my poor wording when describing the nature of my background jobs. When I said "keep the state of execution" I meant to be able to monitor the execution state of the tasks remotely.