Skip to content

feat(rlix): bound the resize chain with lock/RPC timeouts + param validation#35

Open
JunzheJoe wants to merge 1 commit into
zhenyu/miles-mvp-e2efrom
joe/resize-timeouts
Open

feat(rlix): bound the resize chain with lock/RPC timeouts + param validation#35
JunzheJoe wants to merge 1 commit into
zhenyu/miles-mvp-e2efrom
joe/resize-timeouts

Conversation

@JunzheJoe

Copy link
Copy Markdown
Collaborator

What

Review findings F2-design-1/2, F3-TIMEOUT-1/2/3 and S12-RESIZE-NOTIMEOUT (F1-F12 code review). Same problem family as the guide's open MED1 (concurrent-resize under max_concurrency=4): the resize chain had unbounded blocking surfaces at three layers, so one wedged component could hang callers forever — and through the central scheduling loop, stall every pipeline's request/release processing. The fail-fast policy only covers exceptions, not hangs.

1. MilesCoordinator lock timeout (new _resize_locked)

PipelineCoordinator acquires _resize_sync_lock with RLIX_RESIZE_LOCK_TIMEOUT_S (default 180s) and raises on timeout; MilesCoordinator used bare with self._resize_sync_lock: in 9 places. New _resize_locked(op) contextmanager mirrors the same env var and raises with the blocked op's name. All 9 sites converted.

2. Resize-chain ray.get timeouts (MILES_RESIZE_RPC_TIMEOUT_S, default 180s)

shrink_engines / get_engine_states / expand_engines / sync_selected_workers / activate_routing were awaited without timeout. Default leaves margin over the service-internal ROLL_SELECTIVE_MODEL_UPDATE_TIMEOUT_S (150s) and the shrink-side 30s drain. <=0 disables (standard parse_env_timeout_s semantics).

3. Scheduler fan-out timeout (RLIX_RESIZE_RPC_TIMEOUT_S, default 300s)

_execute_resize_calls gathers coordinator resize_infer RPCs outside the scheduler lock with no bound. New _gather_resize_rpcs wraps both fan-outs in asyncio.wait_for; a timeout raises and routes through scheduling_cycle's existing fail-fast shutdown path (consistent with the fail-fast-only operational policy). Default sits above the coordinator-side worst case.

4. resize_infer param validation

Reuses rlix.pipeline.utils.validate_resize_params (exactly one of remove/add non-empty) — same contract as PipelineCoordinator.resize_infer. The scheduler already issues shrinks and expands as separate one-sided RPCs, so a mixed call indicates a caller bug.

Not in scope

The full concurrent-resize stress test stays an M11.3 follow-up (guide MED1); this is the minimal bound-everything version.

Tests

tests/test_miles_coordinator_resize_guards.py:

  • resize_infer rejects mixed shrink+expand and double-empty calls
  • lock acquisition times out with a RuntimeError instead of blocking when the lock is wedged, and succeeds after release

Full suite: 69 passed, 3 skipped (with #33/#34's tests included).

…idation

The resize chain had three unbounded blocking surfaces, so one wedged
component (e.g. a stuck SGLang drain inside shrink_engines) could hang
callers forever — and via the central scheduling loop, stall every
pipeline's request/release processing (guide MED1 family):

1. MilesCoordinator used bare 'with self._resize_sync_lock:' everywhere.
   New _resize_locked(op) contextmanager mirrors PipelineCoordinator's
   RLIX_RESIZE_LOCK_TIMEOUT_S discipline (default 180s) and raises with
   the op name on timeout. All 9 sites converted.

2. The 5 resize-chain ray.get calls into miles (shrink_engines /
   get_engine_states / expand_engines / sync_selected_workers /
   activate_routing) now carry MILES_RESIZE_RPC_TIMEOUT_S (default
   180s, margin over the service-internal 150s; <=0 disables).

3. Scheduler-side _execute_resize_calls wraps both gathers in
   asyncio.wait_for via RLIX_RESIZE_RPC_TIMEOUT_S (default 300s, above
   the coordinator-side worst case). Timeout raises and routes through
   scheduling_cycle's existing fail-fast shutdown path.

Also: resize_infer now validates params via validate_resize_params
(exactly one of remove/add non-empty), matching the
PipelineCoordinator.resize_infer contract; the scheduler already issues
shrinks and expands as separate one-sided RPCs.

The full concurrent-resize stress test remains an M11.3 follow-up; this
is the minimal bound-everything version.
@JunzheJoe JunzheJoe requested a review from taoluo July 7, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant