v1.7.0 — Worker circuit breaker redesign

What's New

A long-running index batch could blow the ChromaDB worker's RPC deadline and latch vector search off permanently. Searches silently degraded to keyword-only behind an HTTP 200 and a healthy container, and stayed that way until someone restarted it. This release makes that state unreachable.

Fixed

• Permanent silent degradation. Replaced the breaker with a locked state machine (CLOSED → OPEN → RECOVERING → CLOSED, plus STOPPED) driven by a single permanent Event-based supervisor. A recovery epoch and worker generation are carried through every transition and health is published only by compare-and-set, so a stale probe cannot overwrite a newer failure. There is no reachable state where vector serving is off and nothing is scheduled to restore it.
• A malformed query no longer degrades search for everyone. Worker errors are classified at the raise site; request/application errors (empty query text, bad collection) never open a backend-health breaker.
• Unbounded ready-wait. _start_worker used a blocking readline() whose deadline was only evaluated between reads, so a worker that never signalled ready blocked the supervisor indefinitely and READY_TIMEOUT had no effect.
• Recovery on the request path. The request path no longer starts or restarts a worker — only the supervisor owns process lifecycle — and query admission fails fast instead of waiting out a ~25s recovery probe.
• Health could be declared while a collection was unloaded. The recovery probe now asserts a per-collection sentinel instead of accepting "did not throw".

Added

• Embedding-model manifest. Collections are stamped with the model that produced their vectors and refused on mismatch. Two same-dimension indexes built by different models previously loaded without error and returned confidently-ranked noise. scripts/stamp_manifest.py migrates indexes created before this check existed.
• GET /health/vector — reports breaker state, epoch, generation, consecutive_failures, next_probe_in_s, state_age_s and served_collections, so a monitor can alert on vector death without the container healthcheck restarting mid-index. Alert on state != CLOSED, and on state_age_s > 300 while RECOVERING.
• New environment variables, all with working defaults: RPC_TIMEOUT_QUERY, RPC_TIMEOUT_INDEX, RPC_TIMEOUT_PING, VECTOR_QUERY_LOCK_TIMEOUT, VECTOR_LIVENESS_POLL_S, VECTOR_PROBE_REQUIRED_COLLECTIONS.

Verification

55 worker-protocol tests, up from 34. Fault-injected against a live container rather than only unit-tested:

• SIGSTOP the worker → search degrades to fts5 with /health/vector 503 → self-heals to hybrid/200 in ~20s, RestartCount=0
• worker held stopped across repeated probes → backoff walks 120/300/900/1800 and still recovers once the fault clears

Upgrade Notes

No configuration change is required. Existing indexes carry no manifest and are served unverified with a warning; run scripts/stamp_manifest.py <persist_dir> <embedding_model> once (with the container stopped) to enable mismatch protection on them.