[Bug]: busy_blocks leaks +1 per concurrent job teardown (lost decrement in JobTerminating pipeline)

[eshaanmoorjani]

Steps to reproduce

1. dstack server 0.20.29 (code path unchanged in 0.21.0), SSH fleet, one host with blocks: 8.
2. Cause several jobs on the same instance to tear down within the same processing window — easiest reliable trigger is a failure storm (submit N tasks that all exit nonzero within seconds, e.g. a bad env var shared by all of them). Batched stops can also do it.
3. Compare instances.busy_blocks with the sum of blocks over jobs still attached to the instance.

Actual behaviour

busy_blocks permanently over-counts by +1 per collision. On our pool it reached +5 (busy_blocks: 7 with two 1-block runs attached), making 5 of 8 blocks unschedulable: the placement gate is idle_blocks = total_blocks - busy_blocks (services/instances.py), so large jobs silently get "no offers". There is no reconciler; the value is only reset at instance creation, so the damage survives server restarts. Before we automated an external repair, the only remedy was deleting and re-creating the fleet.

Root cause

_process_terminating_job computes the new counter as an absolute value from a stale read and stages it in the update map:

# background/pipeline_tasks/jobs_terminating.py (0.20.29, line ~717)
busy_blocks = instance_model.busy_blocks - _get_job_occupied_blocks(jrd)
instance_update_map["busy_blocks"] = busy_blocks

When several jobs of the same instance are processed concurrently, each computes from the same base value and the writes collapse — all but one decrement are lost.

The pre-pipeline path did not have this bug: background/scheduled_tasks/terminating_jobs.py (removed in #3749) locked the instance row (with_for_update(skip_locked=True, key_share=True)) and applied a relative in-place instance_model.busy_blocks -= _get_job_occupied_blocks(jrd) under that lock. #3683 covered the assign-side race and was fixed by the pipelines migration; the unassign side regressed instead.

Evidence — the server's own events table

Every assign/unassign logs Instance blocks: N/M busy, so the counter history can be replayed:

SELECT recorded_at, message FROM events
WHERE message LIKE '%Instance blocks:%' ORDER BY recorded_at;

Two consecutive UNASSIGN events printing the same value = one lost decrement. Our replay for a single day (fresh instance created 06:26, so the counter started at 0) shows exactly five collisions, matching the observed +5 skew to the block:

09:20:37 UNASSIGN -> 5/8
09:20:37 UNASSIGN -> 5/8   <<< lost decrement
09:20:37 UNASSIGN -> 5/8   <<< lost decrement
...
09:28:21 UNASSIGN -> 5/8
09:28:24 UNASSIGN -> 5/8   <<< lost decrement
...
09:29:59 UNASSIGN -> 7/8
09:30:01 UNASSIGN -> 7/8   <<< lost decrement
...
09:31:03 UNASSIGN -> 7/8
09:31:03 UNASSIGN -> 7/8   <<< lost decrement

All five collisions coincide with bursts where 3+ containers exited within the same second. An earlier +2 occurrence on the same pool shows the identical signature.

Expected behaviour

busy_blocks converges to the sum of blocks of jobs attached to the instance. A relative SQL update (busy_blocks = InstanceModel.busy_blocks - :n), re-locking the instance row as the legacy task did, or recomputing from attached jobs at write time would all fix it.

dstack version

0.20.29 (server + CLI). The code shape is unchanged in 0.21.0.

Server logs

The events-table replay above is more precise than the logs; happy to provide full logs or the replay script if useful.

[jvstme]

Duplicate of #3956. Keeping this new ticket for greater visibility

[r4victor]

Thanks for reporting!

The root cause is improper related instance locking on SQLite: the _lock_related_instance must commit before releasing the in-memory lock and it doesn't.