feat(rlix): bound the resize chain with lock/RPC timeouts + param validation#35
Open
JunzheJoe wants to merge 1 commit into
Open
feat(rlix): bound the resize chain with lock/RPC timeouts + param validation#35JunzheJoe wants to merge 1 commit into
JunzheJoe wants to merge 1 commit into
Conversation
…idation The resize chain had three unbounded blocking surfaces, so one wedged component (e.g. a stuck SGLang drain inside shrink_engines) could hang callers forever — and via the central scheduling loop, stall every pipeline's request/release processing (guide MED1 family): 1. MilesCoordinator used bare 'with self._resize_sync_lock:' everywhere. New _resize_locked(op) contextmanager mirrors PipelineCoordinator's RLIX_RESIZE_LOCK_TIMEOUT_S discipline (default 180s) and raises with the op name on timeout. All 9 sites converted. 2. The 5 resize-chain ray.get calls into miles (shrink_engines / get_engine_states / expand_engines / sync_selected_workers / activate_routing) now carry MILES_RESIZE_RPC_TIMEOUT_S (default 180s, margin over the service-internal 150s; <=0 disables). 3. Scheduler-side _execute_resize_calls wraps both gathers in asyncio.wait_for via RLIX_RESIZE_RPC_TIMEOUT_S (default 300s, above the coordinator-side worst case). Timeout raises and routes through scheduling_cycle's existing fail-fast shutdown path. Also: resize_infer now validates params via validate_resize_params (exactly one of remove/add non-empty), matching the PipelineCoordinator.resize_infer contract; the scheduler already issues shrinks and expands as separate one-sided RPCs. The full concurrent-resize stress test remains an M11.3 follow-up; this is the minimal bound-everything version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Review findings F2-design-1/2, F3-TIMEOUT-1/2/3 and S12-RESIZE-NOTIMEOUT (F1-F12 code review). Same problem family as the guide's open MED1 (concurrent-resize under
max_concurrency=4): the resize chain had unbounded blocking surfaces at three layers, so one wedged component could hang callers forever — and through the central scheduling loop, stall every pipeline's request/release processing. The fail-fast policy only covers exceptions, not hangs.1.
MilesCoordinatorlock timeout (new_resize_locked)PipelineCoordinatoracquires_resize_sync_lockwithRLIX_RESIZE_LOCK_TIMEOUT_S(default 180s) and raises on timeout;MilesCoordinatorused barewith self._resize_sync_lock:in 9 places. New_resize_locked(op)contextmanager mirrors the same env var and raises with the blocked op's name. All 9 sites converted.2. Resize-chain
ray.gettimeouts (MILES_RESIZE_RPC_TIMEOUT_S, default 180s)shrink_engines/get_engine_states/expand_engines/sync_selected_workers/activate_routingwere awaited without timeout. Default leaves margin over the service-internalROLL_SELECTIVE_MODEL_UPDATE_TIMEOUT_S(150s) and the shrink-side 30s drain.<=0disables (standardparse_env_timeout_ssemantics).3. Scheduler fan-out timeout (
RLIX_RESIZE_RPC_TIMEOUT_S, default 300s)_execute_resize_callsgathers coordinatorresize_inferRPCs outside the scheduler lock with no bound. New_gather_resize_rpcswraps both fan-outs inasyncio.wait_for; a timeout raises and routes throughscheduling_cycle's existing fail-fast shutdown path (consistent with the fail-fast-only operational policy). Default sits above the coordinator-side worst case.4.
resize_inferparam validationReuses
rlix.pipeline.utils.validate_resize_params(exactly one of remove/add non-empty) — same contract asPipelineCoordinator.resize_infer. The scheduler already issues shrinks and expands as separate one-sided RPCs, so a mixed call indicates a caller bug.Not in scope
The full concurrent-resize stress test stays an M11.3 follow-up (guide MED1); this is the minimal bound-everything version.
Tests
tests/test_miles_coordinator_resize_guards.py:resize_inferrejects mixed shrink+expand and double-empty callsFull suite: 69 passed, 3 skipped (with #33/#34's tests included).