Parallel catchup (v2) false alarm on condition "no worker should be running while queues are empty"

[jayz22]

Here is the assertion.

Apparently it can produce false alarm (such as what we saw recently).

One of the possible reason is that a worker might be temporarily unreachable (either memory starved or coreDNS/ingress/network related issues) but it hasn't failed or crashed. Due to the retry logic another worker might start working on the same job. Eventually one of them finishes and the mission declares all jobs successful. However the other work is still running, so that sanity check fails.

I think the correct fix is to just loosen that check (we still want the retry logic as an extra safety layer) and provide better diagnostics for when such scenarios happen.

[anupsdf]

Is it possible to have some locking mechanism (with expiration) on the job to avoid two workers from working on the same job?

[jayz22]

Is it possible to have some locking mechanism (with expiration) on the job to avoid two workers from working on the same job?

So normally this won't happen at all. The retry mechanism doesn't kick in until the very end, when all workers claim they are free, and there is still jobs left in the in-progress queue.
One much easier thing we could do is just to ping the all worker N times (with interval of 1 min or something) to make sure they are really down, not just temporarily unresponsive. (That acts similarly to a lock with expiration)
We still do want to keep the retry mechanism though if the worker has been unresponsive for too long. Even if they somehow responds later it's just much better for safety to have another worker do the same work since they'll need to wait for everything to finish anyway.

[jacekn]

A possible reason for this is a race condition in the job monitor. Worker status and queue statuses are gathered sequentially so we be ending up in a situation like this:

1. We poll workers status in a loop. Only one worker is catching up and that fact is stored
2. While the worker poll loop is still running the worker finishes catch up, its job is removed from the running queue
3. We get queue status - all queues empty
4. Since we set the worker as "caching up" in step 1 there is a discrepancy

The error condition describe above will only be transient - next job monitor run will correctly mark all workers as down and that will match queue status. So perhaps a simple retry logic with 1min sleep will be enough to avoid the problem.

[jayz22]

A possible reason for this is a race condition in the job monitor. Worker status and queue statuses are gathered sequentially so we be ending up in a situation like this:

Definitely a possibility, and we need to improve the status querying logic and eventually the core endpoint responsiveness for sure.

Also possible if the core info didn't respond in time, the job-monitor treats the worker as "down" and gives its job to a different worker. When the first work finishes, it refreshes the in-progress queue (queue becomes empty) but the 2nd worker is still running. Given that you've observed 1. long response time from core::info 2. some running jobs already have metrics (indicating it's been finished before), I would suggest this is the most likely reason.