Background & scheduled jobs

[davewasmer]

The framework should provide a top level abstraction similar to ActiveJob for Rails, where different job backend can plug into the same public API.

[knownasilya]

Just stumbled on https://github.com/rschmukler/agenda

[acorncom]

Re: the above, looks like it's currently not DB-agnostic (Mongo specific atm) but there is some interest in changing that: agenda/agenda#348 No word from the author on that front though ...

I'd be interested in seeing this fleshed out as I could use it for the help wanted app ...

[davewasmer]

Agenda looks interesting. Might be worth investigating the LOE of adding database adapters to it so we could use it.

In general, my thoughts here are:

• We could take an adapter style approach here (a la Rails ActiveJob), where Denali doesn't know or care how a job is run, it just provides the interface for specifying jobs, and it's up to the user to provide runners.
• The other approach is that we bake in runners as well, so that Denali will schedule and execute the jobs.

Node's asynchronous model makes things interesting - you could potentially do background process in the same process as the web server. Of course, that exposes you to the risk of a bg job blocking the CPU and thus incoming web requests. So supporting in-process bg jobs might be a nice optional feature, but we'll need some non-blocking answer as well.

It looks like agenda doesn't manage a worker pool either, which is something Denali would need to support. In my past research, I didn't find any single solution for node bg jobs that satisfied everything, but there might be a way we can leverage something like Agenda to get us 70% there.

[davewasmer]

Closing in favor of http://discuss.denalijs.org/t/background-scheduled-jobs/27