Skip to content

OME holds engine_sem across the whole retry chain, now including backoff sleeps #396

Description

@Kendrick-Song

Runner.run acquires engine_sem for the entire retry chain — see the module docstring in infra/ome/_dispatch/runner.py ("Holds engine_sem for the full retry chain so concurrency cap applies end-to-end"). That predates PR #393.

#393 added exponential backoff between attempts (1s → 2s → 4s, cap 10s, plus jitter). Those sleeps now happen inside the semaphore, so a strategy with max_retries=3 occupies a concurrency slot for up to ~8.5s where it previously held it for milliseconds.

Under max_concurrent_runs = 20 (default), a systemic failure — which is exactly the scenario backoff exists for — now ties up slots for seconds each. That makes the 300s drain caps at memory/cascade/_backfill.py:885 and :1261 materially easier to hit.

Verified during the #393 review that the sleep holds no domain lock: _ClusterMissingError propagates out of async with get_partition_lock(...) (extract_agent_skill.py:161) before the sleep, and no RUNNING row is left behind, so crash recovery is unaffected. The concern is purely concurrency-slot occupancy.

Options: release the semaphore around the sleep and re-acquire; move backoff to the scheduler (re-enqueue with a delayed run_date instead of sleeping in-process); or size max_concurrent_runs with the new occupancy in mind and document it.

Note the backoff defaults are also oversized for their only remaining caller — see the note in #393's description.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions