fix(dispatch): recover slot after silent Lemonade eviction

[thinmintdev]

Summary

Chats returned 502 dispatch.upstream_unavailable when Lemonade
silently evicted a slot's child (idle/OOM) but hal0's in-memory state
still said the slot was READY/SERVING/IDLE. The swap-window gate from
#385 only catches known non-ready states; the state-drift window
bypasses it entirely and lands on a dead port.

User-reported trace:

chat-completions 502: {"error":{"code":"dispatch.upstream_unavailable",
  "details":{"upstream":"primary",
             "target":"http://127.0.0.1:8001/v1/chat/completions",
             "error":"ConnectError"}}}

Fix

Two pieces:

• src/hal0/slots/manager.py — new SlotManager.recover_evicted_slot()
  drives a full unload + load cycle. Unload forces lemond to drop its
  loaded[] entry (a bare /v1/load returns success without
  re-spawning when loaded[] still claims a model on a dead PID); load
  then re-spawns the child. Unload failure is logged but doesn't bail —
  load may still succeed.

• src/hal0/dispatcher/router.py — _forward_direct and
  _forward_streaming retry once after a recoverable transport
  error from a slot upstream. Recoverable set:

  • httpx.ConnectError — TCP refused before request
  • httpx.RemoteProtocolError — peer dropped mid-request (the
    eviction-race signature)

  Read/write timeouts are intentionally excluded — those usually mean
  overload, not death. Attempt-1 cap prevents infinite loops; recovery
  itself raising falls back to the original UpstreamUnavailable
  envelope so the error trail stays useful.

Why not change the dispatch target instead?

Routing the data plane through lemond's /api/v1/chat/completions
(port 13305) was considered but rejected — both upstream Lemonade and
hal0/lemonade/client.py:5-6 explicitly call out the control-plane /
data-plane split as a perf design decision. This fix preserves that
boundary and adds lazy recovery at the existing seam.

Test coverage

Six new dispatcher tests in tests/dispatcher/test_serving_integration.py:

• test_forward_recovers_from_silent_eviction_and_retries — happy path
• test_forward_gives_up_when_retry_after_recovery_still_fails — no infinite loop
• test_forward_remote_connect_error_does_not_attempt_recovery — remote upstreams skip recovery
• test_forward_streaming_recovers_from_silent_eviction — streaming gets same treatment
• test_forward_recovers_from_remote_protocol_error — peer-dropped variant
• test_forward_gives_up_when_recover_evicted_slot_itself_raises — recover failure surfaces original error

215 dispatcher + slots tests pass; ruff check + format clean.

LXC verification

Recovery branch confirmed end-to-end on hal0 LXC under the
orphan-process trigger:

dispatch.upstream_dead_attempting_recover  slot=primary
slot.recover_evicted_dispatched
lemonade.provider.unload
lemonade.provider.load
dispatch.upstream_recovered

Known limitation (out of scope)

If lemond re-spawns the child on a port other than the one hal0's slot
config expects, the retry's target_url is stale and still hits the
dead port. Root cause: LemonadeProvider.load doesn't pass a port
hint to /v1/load, and hal0 never reads backend_url back from
/v1/health. Separate issue; logged for follow-up.

Related

• fix(dispatch): gate slot forwards during swap window with structured 503 #385 — swap-window gate (complementary). Together they cover the
  full eviction lifecycle: gate catches known-not-ready, this PR
  catches state-drift.
• ADR-0008 — Lemonade integration boundary.

Test plan

• pytest tests/dispatcher/ tests/slots/
• ruff check + ruff format --check
• LXC smoke: trace recovery branch firing under orphan-process repro
• Production validation: monitor dispatch.upstream_dead_attempting_recover rate over the next week to verify it catches real silent evictions

🤖 Generated with Claude Code