feat(coordination): add cross-instance Coordinator for multi-instance consistency

[magiccao]

Description

Add a Coordinator abstraction that unifies several process-local singletons
(EmbeddingTaskTracker, RequestWaitTracker, SemanticQueue coalesce
versions, request stats) behind a shared backend, enabling consistent state
across multiple load-balanced server instances.

Distributed Fault-Tolerance Design

Single-instance deployments are unaffected (the default memory backend is a
thin wrapper over existing dict/lock patterns). For Redis-backed deployments, a
layered self-healing approach prevents queuefs event loss:

1. Owner-side completion poller (EmbeddingTaskTracker)

Under the distributed backend, the final decrement() may land on a
non-owner instance that holds neither the callback nor the event loop it
must run on. The registering instance launches a background poller that
watches the shared remaining counter in the Coordinator. When the counter
reaches zero — regardless of which instance decremented it — the owner
fires the completion callback locally. The poller also holds the deadline
for the distributed-only watchdog timeout, triggering on_timeout for
graceful self-healing when embedding tasks stall.

2. TTL refresh (_touch_reg)

All Coordinator keys carry a configurable TTL so that a crash or restart
does not leave orphaned state. Long-running tasks extend their key TTLs
via a throttled _touch_reg call (refresh interval = ttl_sec / 4)
to prevent premature expiry without generating unbounded Redis traffic.

3. Exponential backoff on transient coordinator errors

Both EmbeddingTaskTracker._watch_until_resolved and
RequestWaitTracker.wait_for_request absorb transient Redis errors and
retry with exponential backoff (0.25 s → 0.5 s → 1.0 s → 2.0 s cap) rather
than propagating exceptions to callers or tight-looping under failure.

4. Two-phase request wait with grace period (RequestWaitTracker)

wait_for_request splits into a processing gate (waits for pending sets to
empty) and a status retrieval phase. On transition to the status phase a
2-second grace window is added on top of any remaining deadline, ensuring
the final queue-status snapshot is readable even when the original timeout
is almost exhausted.

5. Graceful cleanup degradation

cleanup() under the distributed backend catches deletion errors and falls
back to TTL-based expiry with a warning log, avoiding cleanup failures from
blocking the request path.

---

Configuration

Coordination is configured under the storage.coordination key in ov.conf.
The default memory backend requires no additional configuration.

Redis backend

"storage": {
  "coordination": {
    "backend": "redis",
    "redis": {
      "dsn": "redis://redis:6379/0",
      "key_prefix": "ov:coord:",
      "ttl_sec": 3600
    },
    "embedding_completion_timeout_sec": 1800
  }
}

The redis.dsn field can also be omitted and supplied via the
OPENVIKING_COORD_DSN environment variable (preferred for credential
management).

Custom backend

Set backend to a fully-qualified dotted class path. The class must expose a
from_config(cfg: CoordinationConfig) -> Coordinator classmethod. Arbitrary
extra parameters can be passed through custom_params.

"storage": {
  "coordination": {
    "backend": "mycompany.coordination.CustomCoordinator",
    "custom_params": {
      "option_a": "value_a",
      "option_b": 42
    },
    "embedding_completion_timeout_sec": 1800
  }
}

Key options

Field	Default	Description
backend	"memory"	"memory", "redis", or dotted class path
redis.dsn	env OPENVIKING_COORD_DSN	Redis connection string
redis.key_prefix	"ov:coord:"	Key namespace for tenant/cluster isolation
redis.ttl_sec	3600	Key TTL; 0 disables expiry
embedding_completion_timeout_sec	1800	Distributed-only embedding watchdog (seconds)
custom_params	{}	Passed to from_config() for third-party backends

Related Issue

N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit ea14930)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🏅 Score: 90

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.

[github-actions]

Persistent review updated to latest commit ea14930

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.