Add support for background jobs

[hdoordt]

Add support for running jobs in the background

[rustworthy]

Hi folks!

You got something specific on your mind here? I mean something existing feature-rich, something in-house and simplistic (e.g. interval runner crate as part of the workspace)? Or maybe there is machinery you guys create again and again in your projects and it can be packaged as a crate or has already been packaged? I am guessing coordination across multiple app instances should be present anyway (if pg backed - then, say, via locking and skipping locked)?

I was planning to create a simplistic interval runner solution with task_name and last_ran props and only supporting sqlx and PostgreSQL (which - at least from my experience - is just enough for most use cases), but then I found this issue and decided to check in and ask about your experience and plans.

Please let me know if this issue still needs to be addressed and also if you guys have discord (or something similar) or maybe there should be an item in gh discussions or we are already in the right place to discuss the details.

Cheers,
Pavel

cc @hdoordt
cc @marcoow

[marcoow]

We don't really have any specific ideas here so happy to discuss proposals. I'd say the simpler the better

[rustworthy]

I really love what they've done here in woddle and thought maybe having something similar (tasks spawned in-process in each app instance, a two-column PostgreSQL table used to check if it's time to run a task but also to coordinate: so that this or that task is run only by one instance). The code could live in a dedicated crate in the workspace, say, task-runner crate, but the migration will belong to the db crate. Task configs can be provided via the config files, obviously. The tasks themselves can be anything that implements Task or, say, Runnable trait and in nature is very similar to what Luca has here.

Or were you planning to integrate something from the ecosystem? I know a tool exists called tokio-cron-scheduler - it uses tokio-postgres under the hood, but i am not sure it offers coordination across multiple app instances. There is also a tool called apalis-postgres, looks like it's feature-rich. I am pretty sure there are more, but i've never worked with any of those. Instead, i had experience with faktory - but it requires deployment and its Community Edition does not include job deduplication.

I'd go with the first approach, but wdyt?