Design of new queueworker

[pbiggar]

Goals:

• to move Queue scheduling out of the DB, as it causes operational concerns
• to remove the queue-scheduler daemon as it's written in rust and we'd like to not have any rust left in our codebase

Evaluation of Google Pub Sub

Pub-sub does not seem to be a direct match for our queues, for a number of reasons:

• only stores items for a week, we store them for longer (we might need to change this, but not necessarily now)
• does not have a concept of "paused" or "blocked"

So this implies that pub-sub is not necessarily great as the backing store. But perhaps it will be ok as a scheduler.

How would it work for scheduling:

So the idea here is that we have the queue do the scheduling instead of a having a queue-scheduler service. This would mean we're not doing queries for "what's next" on the DB, we'd let the queue do that.

enqueue

Save entry in the DB. Send id to the queue.

dequeue

fetch item from the queue. Fetch event and some metadata from the DB for this event ID. (so it's using the DB as a k/v store for the data and metadata)

put back

• Don't put back for missing
• Don't put back for incomplete
• For error fetching the canvas, increment retries. Reject it if retries not high enough, or acknowledge it if we're not going to retry again. Pub/Sub supports retries and delays.
• For error running the canvas, increment retries
• For errors due to shutdown before acknowledgement, it will just run it again later.

Block / pause

• Check the rule when you load the canvas. If it's blocked or paused, acknowledge to remove it from the queue, and don't change the status.

Unblock, unpause:

Fetch the items and put them in the queue. this happens fairly rarely so the fact that we have to search isn't a big deal.

• TODO: what happens if the same item is added more than once (eg the users toggles this back and forth a bunch)? Can the queue disambiguate? Otherwise the worker needs to set a lock. Needs to be solved. We do not want an actual DB lock though. We could set the row to a value with an expiry time. Could use an ordering key? then when it arrives the status will already be done?

[pbiggar]

So we treat getting an event from the queue as a notification to try to run the event again. The state is tracked by the presence of the event and the fields: lock, retry_count.

    %% Happy path
    Queue[Queue] -->|Receive notification| EventLoad
    EventLoad --> |EventPresent| PauseCheck
    PauseCheck -->|PauseNone| DelayCheck
    DelayCheck -->|DelayNone| LockCheck
    LockCheck -->|LockNone| LockClaim
    LockCheck -->|LockExpired| LockClaim
    LockClaim --> |Claimed| Process
    Process --> |ExecutesToCompletion| Delete
    Delete --> End

    %% Error checking

    %% Done, so don't requeue
    EventLoad -->|EventMissing| End 

    %% Might need to try again if the locked item fails
    LockCheck -->|IsLocked| Queue

    %% We got this too early
    DelayCheck -->|DelayNotYet| Queue

    %% Drop the notification, we'll reload it upon unpause
    PauseCheck -->|Paused| End 
    PauseCheck -->|Blocked| End

    %% Probably another event beside us, no harm requeuing
    LockClaim --> |LockClaimFailed| Queue

    %% Allow retries
    Process --> |Exception| RetryCheck
    RetryCheck --> |RetryAllowed: increment, delay| Queue

    %% That's enough, we're done
    RetryCheck --> |RetryTooMany| Delete
    %% Will probably be dropped in EventMissing
    RetryCheck --> |Exception| Queue

graph LR
    %% Happy path
    Queue[Queue] -->|Receive notification| EventLoad
    EventLoad --> |EventPresent| PauseCheck
    PauseCheck -->|PauseNone| DelayCheck
    DelayCheck -->|DelayNone| LockCheck
    LockCheck -->|LockNone| LockClaim
    LockCheck -->|LockExpired| LockClaim
    LockClaim --> |Claimed| Process
    Process --> |ExecutesToCompletion| Delete
    Delete --> End

    %% Error checking
    EventLoad -->|EventMissing| End
    LockCheck -->|IsLocked| Queue
    PauseCheck -->|Paused| End
    PauseCheck -->|Blocked| End
    DelayCheck -->|DelayNotYet| Queue
    LockClaim --> |LockClaimFailed| Queue
    Process --> |Exception| RetryCheck
    RetryCheck --> |RetryTooMany| Delete
    RetryCheck --> |Exception| Queue
    RetryCheck --> |RetryAllowed: increment, delay| Queue

Loading

[pbiggar]

Before I forget, a major goal was not to use a DB lock for scheduling or blocking access, as we've been having issues from holding that lock open (in ocaml, led to bizarre performance behaviour; in fsharp, it timed out too early)

[StachuDotNet]

I'm not understanding something here. Let me know where I go wrong below:

• We have a Queue (Q), a Queue Scheduler (QS), and a Queue Worker (QW).
• Q is currently a table of Events that get created by emits in BwdServer, and by Crons (via cronchecker)
• QS is in charge of flipping new events to scheduled
• QW is in charge of executing scheduled events, and handling either success (by marking as done) or failure (set back to new and potentially increase retries)
• QS currently only exists in Rust
• QW exists now in both OCaml and F#, but the F# version is ... WIP/incomplete/disabled
• Those Events (described in docs/production/events.md) live in a DB table
• Q has both _meta_data and scheduling data, and the scheduling data being in the DB is our issue
• So, we'd like to remove the scheduling (data? or just process) from the DB, instead handling that externally
• Google Pub/Sub doesn't seem an ideal solution for QW, but may be appropriate for QS

Under enqueue,

Save entry in the DB. Send id to the queue.

I don't understand how we can enqueue only the IDs. The new QS process still has to fetch the scheduling data?
If events (their IDs) are immediately pushed to Pub/Sub, won't the ones with a delay_until value be there inappropriately?

[pbiggar]

To be clear, you're describing the current system/problem.

We have a Queue (Q), a Queue Scheduler (QS), and a Queue Worker (QW).
Q is currently a table of Events that get created by emits in BwdServer, and by Crons (via cronchecker)
QS is in charge of flipping new events to scheduled
QW is in charge of executing scheduled events, and handling either success (by marking as done) or failure (set back to new and potentially increase retries)
QS currently only exists in Rust
Those Events (described in docs/production/events.md) live in a DB table

So far so good.

QW exists now in both OCaml and F#, but the F# version is ... WIP/incomplete/disabled

Incomplete and soon to be disabled.

Q has both _meta_data and scheduling data, and the scheduling data being in the DB is our issue

Yes, that's right. The issue is that handling the scheduling eats about 50% of our DB's CPU right now, and also holds long-running locks in the workers (causes other issues, honestly unsure of the extent of them).

So, we'd like to remove the scheduling (data? or just process) from the DB, instead handling that externally
Google Pub/Sub doesn't seem an ideal solution for QW, but may be appropriate for QS

Right. Pub/Sub just delivers events, it doesn't do compute.

[pbiggar]

Under enqueue,

Save entry in the DB. Send id to the queue.

I don't understand how we can enqueue only the IDs. The new QS process still has to fetch the scheduling data?

We enqueue the ID and the canvasID. That's enough for the QueueWorker to fetch the rest of the info from the DB (we could also encode more than that, but we'd still need to go to the DB for other info).

If events (their IDs) are immediately pushed to Pub/Sub, won't the ones with a delay_until value be there inappropriately?

PubSub supports delays - if we set a delay it won't try to schedule it until the appropriate time (so long as it's within a week). However, it would work even if we didn't - it would just bounce around a lot until we're ready for it.

[pbiggar]

I think the docs didn't explain what this was trying to do in enough detail, so I added more info: https://github.com/darklang/dark/blob/6b8226d59aea6b9af06217a49003eccdaae4678e/docs/eventsV2.md