fix: authoritative + bidirectional liveness reconcile for external docker changes (#1015, #1016)

[milkway]

What & why

Fixes #1015 and #1016 — two bugs with one root cause: the scaler's runtime
liveness reconcile was one-way and treated a container's absence from
list_managed_containers() as "not stopped".

• Auto-recuperação falha quando container gerenciado é removido via Docker CLI (docker rm -f) #1015 — docker rm -f <ruscker-container> leaves the replica Ready in the
  registry forever (absence ≠ stopped), so routing/sticky keeps hitting a dead
  upstream → 502 upstream error, and the phantom still counts toward
  max-replicas, blocking a substitute.
• External docker restart can orphan a running container because runtime reconciliation is one-way #1016 — docker restart <ruscker-container> races the scaler tick: a tick
  that catches the container stopped prunes the replica + drops sessions, then
  Docker finishes the restart and the same container is running again but is
  never re-adopted (runtime re-adoption only happened at Ruscker startup) → an
  orphan invisible to proxy/dashboard/scaler, and a later access can spawn a
  duplicate past max-replicas.

Approach — authoritative, bidirectional reconcile

New backend contract in ruscker-core:

• enum ReplicaLiveness { Running, Stopped, Missing, Unknown } + a
  ContainerBackend::replica_liveness(&[ReplicaLivenessQuery]) -> ReplicaLivenessReport
  that classifies the caller's known replicas and returns running,
  Ruscker-labelled containers for re-adoption. The default impl is fail-safe:
  it reports every replica Unknown and discovers nothing, so mocks / non-Docker
  backends can never trigger a false prune (preserves today's behavior).
• ContainerBackend::wait_until_ready(&Replica) — fail-closed default; re-adoption
  reuses the same readiness check as spawn before publishing a replica Ready.

The key invariant — Missing (authoritative) vs Unknown:

• Local backend: a successful listing means any known container id that's absent is
  authoritatively Missing. A whole-call daemon error → the pass is skipped (as
  before).
• Multi-host: built on the existing HostBatch { reported, failed_hosts } /
  placement machinery. A replica whose owning host answered without it is
  Missing; a replica whose owning host failed/timed out (or an unplaced
  replica while any host is unreachable) is Unknown — never pruned, so a degraded
  fan-out can't forget a live container and spawn duplicates.

Scaler (tick):

• Prunes Missing and Stopped-past-grace through one shared cleanup routine
  (cleanup_replica_state: registry + sessions + metrics-cache + in-flight gauge),
  which stop_one now also uses so those states can't drift apart. ReplicaDown
  alerts/logs name the reason (missing/removed vs restart timed out).
• Models an external restart as a transient state: on first Stopped
  observation the replica is marked non-routable (mark_restarting) and held for a
  short grace window (EXTERNAL_RESTART_GRACE_SECS = 30). It keeps its registry
  slot, so min-replicas stays satisfied and no duplicate is spawned during a
  normal restart. If the same container returns Running it's re-adopted
  (after wait_until_ready); if the grace window elapses it's pruned + replaced.
• Re-adoption respects max-replicas under the spawn-coalescer lock (a running
  orphan beyond the cap is removed rather than exceeding max or duplicating).

Out of scope (fast-follow)

Per the issues' "optional" sections, this PR is the reconcile core only. Not
included: the reactive proxy-path recovery (confirm backend state on an upstream
transport failure + GET/HEAD replay) and the Docker die/destroy/start event
listener. The transient 502 during an in-progress restart is acknowledged as
self-recovering by #1016; the periodic reconcile is the fallback either way.

Tests

• core/docker unit: local absence-after-successful-listing → Missing;
  multi-host absence on a responding host → Missing, replica on a failed host →
  Unknown.
• scaler unit: Missing prunes + drop_replica; Unknown keeps replica &
  sessions & blocks a duplicate spawn; Running → Stopped → Running on the same id
  readopts exactly once (one readiness check, zero spawns); Stopped past the grace
  window is pruned + replaced; restart within grace does not duplicate-spawn
  (locks the External docker restart can orphan a running container because runtime reconciliation is one-way #1016 race). Existing liveness_reconcile_prunes_only_stopped_containers
  stays green.
• docker-it (real daemon): docker rm -f → Missing + clean replacement (no
  phantom blocks/duplicates); docker restart → the same labelled container is
  rediscovered Running, passes readiness, and re-enters the pool with no duplicate.

Gate

cargo test · cargo clippy --all-targets --all-features -D warnings ·
./scripts/i18n-check.sh · cargo test -p ruscker-docker --features docker-it
(48 passed against local Docker 29.6.1) — all green.

🤖 Generated with Claude Code

[milkway]

Review (adversarial pass) — MERGE-OK

Reviewed the seven highest-risk areas directly against the code (an independent codex reviewer was queued but its account hit its usage cap, so this is a manual pass; CI is fully green).

• Multi-host false-prune: classify_multihost_liveness returns Missing only when the owning host is in reported and the container id is absent. A host in failed_hosts, a host no longer connected, or an unplaced replica while any host failed → Unknown (never pruned). Cannot mis-prune a live container behind an unreachable daemon.
• max-replicas: readopt_running_replica and spawn_one serialize on the same state.spawn_locks.entry(spec.id) mutex and both count replicas_of().len() over all states → coherent cap; a running orphan beyond max is removed rather than duplicated.
• Grace window: count = snap.len() counts the Stopped (restarting) replica, so min-replicas doesn't duplicate-spawn during a normal restart (locked by restart_within_grace_does_not_duplicate_spawn). mark_restarting resetting started_at=now makes the replica "young", so the YOUNG_REPLICA_GRACE_SECS idle-tick exemption protects it from premature scale-down; readopt restores the real container-created time.
• Lock order is always spawn_locks → replicas (same as spawn_one) — no inversion. wait_until_ready runs without the registry lock held, so a slow readiness wait can't stall the proxy hot path.
• cleanup_replica_state (now also backing stop_one) only adds metrics/in-flight removal — strictly more cleanup, no behavior regression. inflight_gc reading the post-reconcile registry is correct. No unwraps on backend data in the tick path.

Accepted trade-off (non-blocking): a re-adoption's readiness wait holds the per-spec spawn lock for its duration (bounded by the readiness timeout), which correctly blocks a concurrent duplicate spawn.

Gate: cargo test, clippy -D warnings, i18n parity, and docker-it (48 passed against real Docker 29.6.1) all green.