Runner.run acquires engine_sem for the entire retry chain — see the module docstring in infra/ome/_dispatch/runner.py ("Holds engine_sem for the full retry chain so concurrency cap applies end-to-end"). That predates PR #393.
#393 added exponential backoff between attempts (1s → 2s → 4s, cap 10s, plus jitter). Those sleeps now happen inside the semaphore, so a strategy with max_retries=3 occupies a concurrency slot for up to ~8.5s where it previously held it for milliseconds.
Under max_concurrent_runs = 20 (default), a systemic failure — which is exactly the scenario backoff exists for — now ties up slots for seconds each. That makes the 300s drain caps at memory/cascade/_backfill.py:885 and :1261 materially easier to hit.
Verified during the #393 review that the sleep holds no domain lock: _ClusterMissingError propagates out of async with get_partition_lock(...) (extract_agent_skill.py:161) before the sleep, and no RUNNING row is left behind, so crash recovery is unaffected. The concern is purely concurrency-slot occupancy.
Options: release the semaphore around the sleep and re-acquire; move backoff to the scheduler (re-enqueue with a delayed run_date instead of sleeping in-process); or size max_concurrent_runs with the new occupancy in mind and document it.
Note the backoff defaults are also oversized for their only remaining caller — see the note in #393's description.
Runner.runacquiresengine_semfor the entire retry chain — see the module docstring ininfra/ome/_dispatch/runner.py("Holds engine_sem for the full retry chain so concurrency cap applies end-to-end"). That predates PR #393.#393 added exponential backoff between attempts (
1s → 2s → 4s, cap 10s, plus jitter). Those sleeps now happen inside the semaphore, so a strategy withmax_retries=3occupies a concurrency slot for up to ~8.5s where it previously held it for milliseconds.Under
max_concurrent_runs = 20(default), a systemic failure — which is exactly the scenario backoff exists for — now ties up slots for seconds each. That makes the 300s drain caps atmemory/cascade/_backfill.py:885and:1261materially easier to hit.Verified during the #393 review that the sleep holds no domain lock:
_ClusterMissingErrorpropagates out ofasync with get_partition_lock(...)(extract_agent_skill.py:161) before the sleep, and noRUNNINGrow is left behind, so crash recovery is unaffected. The concern is purely concurrency-slot occupancy.Options: release the semaphore around the sleep and re-acquire; move backoff to the scheduler (re-enqueue with a delayed
run_dateinstead of sleeping in-process); or sizemax_concurrent_runswith the new occupancy in mind and document it.Note the backoff defaults are also oversized for their only remaining caller — see the note in #393's description.