RFC for CouchDB background workers

[nickva]

No description provided.

[garrensmith]

Nice start @nickva thanks for starting this

[sansato]

This sounds Very good! Thank you very much.

Is the data model for this lightweight to use with good API and manage progress/backlog/and so for "middle-tier" tasks? Think of use-cases being served in old style database with asynchronous triggers following API calls, to change paradigm, like Kafka with consumer-groups. Software agents can connect to couchdb's job-service api, claim ready to serve shard of queue for specific application-layer job, get change-stream like replication queue and post notifications if needed to show progress. It can be simpler and get good help from job infrastructure, make business infrastructure easier to control and manage. They enjoy most of same things like indexing does, optional "reprocess whole database", live processing of newest records even during rebuild and so.

[nickva]

@sansato

Is the data model for this lightweight to use with good API and manage progress/backlog/and so for "middle-tier" tasks?

Ah, that's an interesting idea. The background jobs systems could in theory accept jobs from the outside if there was an HTTP API wrapper around it. So far in the design, jobs's keys and values are FDB tuples and/or json encoded objects so it seems doable. Perhaps something for a future iteration...

We also mention that workers could be implemented as any process which can connect to the FDB cluster. Generally it will be Erlang processes (or applications) but those workers could be written in other language as long as they can talk to the FDB cluster. And we might even end up with that when implementing some of the secondary indexing work. Having a completely custom set of workers, and a more fleshed out API for the workers to claim and do work would be possible. But I suggest we leave that for the future as well.

[sansato]

for now, could highlight the integration spots to decrease odds that "future" resolves to "never".

Thanks!

[nickva]

Thank you all for the feedback. Taking it into consideration, and also
discussing with @davisp I think it would make sense make these changes:

1. Switch the focus from workers to jobs. Previously the focus was on workers.
   How many workers are running, how they are registered, monitoring their
   liveness, etc. The concept of worker was similar to a job manager or job
   supervisor. That introduced unecessary complexity. It was also confusing
   that one worker was running more than one job. So instead the API and the
   implementation switch to making a job as central concept. Users add jobs,
   resubmit jobs, etc. A worker now become just what function (or process)
   happens to execute the job at a particular instant

2. Previously there was no clear distinction between how the framework would be
   implemented vs what framework API would look like. In the current version,
   the main part of the RFC focuses on the framework API and then implementation
   is discussed separately.

3. Some concepts like "health" were removed and instead re-used "activity"
   since it is already something CouchDB knows about (thanks @janl for pointing
   it out).

4. Based on indexing requirements, introduced the concept of job-resubmission.

5. Mutual exclusion guarantees will be important for indexing and so there is
   a section discussing that aspect as well.

[garrensmith]

This is really nice and much clearer than the original worker rfc. Nice work.

[davisp]

Super awesome work on this! I'm quite like how the overall structure has turned out and made the whole approach seem a lot more intuitive.

I would like to spend some effort reorganizing the layout of this doc though. I could see this either being the basis for or even just as-is the documentation for a developer that wants to implement a new job type. As is that developer would have to read the entire document and comprehend the entire structure and implementation of job processing before being certain that a job processing implementation was correct. I think splitting it out into a few more sections that go from high level to progressively lower level details would be quite helpful for future when we've paged this out of brain cache and accidentally overwritten those meat sectors with intervening years of coding. I'd propose something like the following:

1. Front matter
2. Reason for existence
3. High-level description of a job
4. API description from the developer point of view
5. Internals concepts
6. API algorithm descriptions
7. Explicit data model
8. API implementation referencing fdb level functions and data model
9. Mutual exclusion (if not sufficiently covered in 8)
10. Activity monitor (samesies if not covered)

All of those things already exist, but they're jumbled around in such a way that makes the various concepts hard to absorb. I tend to think of it as starting high level and then working towards actual bytes. I'm basically thinking that sections 3 and 4 should basically contain enough information for a developer implementing a new job type and then everything else just gets progressively closer to the actual fdb interface.

Again, super awesome work!

[nickva]

@davisp, @garrensmith Thanks for the great comments and input. I had pushed the update that hopefully addresses most of the issues. I think the API looks cleaner and the structure of the RFC is more reasonable as well.

[garrensmith]

Some minor changes. But really nice RFC. Really nice and clear.

[garrensmith]

Should we mention that a finished job could be resubmitted later?

[nickva]

If the job was running, then resubmitting it would make it pending again after the job processor calls finish. However if it already finished, then it won't start running again.

[nickva]

It was updated such that job creator can just call add/4,5 again. If the job is running it will be flagged to be re-enqueued when it finishes. If it is pending, only the scheduled time value may be updated (if provided). If the job was already finished it is re-enqueued for execution.

[kocolosk]

@davisp @garrensmith @nickva is this one approved and ready to merge? I don't see any outstanding comments

[nickva]

I think I addressed most of the issues. Didn't know if @davisp wanted to take another look at it

[davisp]

I haven't got anything to add.