Engineering state store

Shared State Store Scenario

The shared state store is KeepPeek's durable coordination plane. Clients, services, and coordinators publish small desired-state documents; other authorized participants watch those documents and use ordinary media commands to make the desired state real.

It is deliberately not a media bus, recording database, credential store, or replacement for ServerCapabilities. State asks for an outcome. Capabilities and publication/subscription results report whether that outcome is actually available.

Core model

A state value is one complete StateEntry identified by (namespace, key). KeepPeek assigns the owner identity, revision, timestamps, and expiration. Writers replace complete documents rather than mutating individual fields, so every watch update is independently usable.

Concept	Meaning
Namespace	Authorization and lifecycle boundary, such as system/, service/, group/, or a private user/device namespace.
Key	Stable path-like name within a namespace. One orchestration intent uses one key.
Schema	Versioned document type, such as keeppeek.media-intent.v1.
Revision	Monotonically increasing namespace revision used for compare-and-set and ordered watches.
Owner	Server-issued principal identity permitted to write the entry under namespace policy.
TTL	Optional liveness lease. Expiry deletes the entry and is observable as EXPIRE.

State documents are bounded structured data. They must not contain media bytes, JPEGs, access keys, cookies, passwords, raw SDP, or arbitrary logs. Large or historical data belongs in a dedicated store; live media belongs in RTP or media-data bindings.

Namespaces and authorization

KeepPeek applies the following namespace policy to the authenticated principal. Namespace spelling does not confer service identity or group membership:

Namespace	Typical writer	Typical readers	Use
system/	No client writes	Administrators	Server-owned observed coordination state
service/<service-id>/	Administrators	Administrators	Worker liveness, leases, output intent
group/<group-id>/	Administrators	Administrators	Shared group preferences and non-authoritative UI state
user/<owner-id>/	Authenticated owner or administrator	Owner or administrator	Personal desired subscriptions and local preferences

Service-specific grants and group-member grants are not implemented. Non-administrators fail closed for these namespaces, including watch delivery. KeepPeek assigns the authenticated principal as the owner; a client cannot supply another owner identity.

Read, write, and delete

GetState returns the complete current entry. PutState atomically replaces it. DeleteState removes it. A missing key returns STATE_STORE_ERROR_CODE_NOT_FOUND.

Writers use expected_revision to avoid lost updates:

expected_revision	Meaning
Absent	Blind replacement permitted by namespace policy
0	Create only; reject if the key already exists
Nonzero	Replace or delete only when it equals the current entry revision

On a mismatch, KeepPeek returns STATE_STORE_ERROR_CODE_CONFLICT and the current revision. The writer rereads, merges its own domain-specific intent, and retries. A client never retries a blind write after a conflict because that would discard another writer's state.

An optional nonzero TTL requests a liveness lease. A value below one second is rejected with STATE_STORE_ERROR_CODE_TTL_INVALID; a value above 24 hours is clamped to 24 hours, and the entry carries the accepted expires_at_ms. Expiry is inclusive: an entry whose deadline equals the current time already reads as not found. KeepPeek emits an EXPIRE update when an accepted TTL elapses. Services refresh their own lease with compare-and-set writes; they do not refresh another owner's entry. Restart purges overdue leases before serving: expired rows are deleted, their bytes leave the counter, and each affected namespace revision advances by one per expired entry, exactly as if the leases had expired lazily, so later writes stay monotonic. MQTT integration state and settings-backed entries are live configuration rather than leases and reject any TTL.

Watches without a snapshot gap

WatchState atomically registers a watch and captures all current matching keys. Its StateStoreResult.watch response contains the complete initial snapshot exactly as it existed at snapshot_revision. Later updates follow that snapshot as StateStoreWatchUpdate notifications on the reliable ordered control channel, each carrying a per-watch watch_sequence starting at 1. Settings-backed namespaces follow the same lifecycle: the snapshot is captured while holding the configuration lock so no committed mutation can slip between the snapshot and the registration, and only file-committed puts and deletes fan out to watchers. Settings entries never expire, so their watches carry no EXPIRE updates.

sequenceDiagram
    participant C as Coordinator
    participant K as KeepPeek state store

    C->>K: WatchState { request_id: 41, watch_id: transcode-intents, namespace: service/transcoder-a, key_prefix: intents/ }
    K-->>C: StateStoreResult { request_id: 41, snapshot_revision: 120, entries }
    K-->>C: StateStoreWatchUpdate { watch_sequence: 1, revision: 121, kind: PUT, entry }
    C->>K: WatchStateAck { request_id: 42, watch_id: transcode-intents, applied_sequence: 1 }
    K-->>C: StateWatchAckResult { request_id: 42, watch_id: transcode-intents, applied_sequence: 1 }
    K-->>C: StateStoreWatchUpdate { watch_sequence: 2, revision: 122, kind: EXPIRE, key }

Loading

A client installs the snapshot first, then applies an update only when its watch_sequence is exactly one above the highest applied. The namespace revision also covers keys outside a prefix filter, so a revision skip without a sequence skip is harmless, while a sequence skip is a gap. A sequence gap, StateStoreWatchClosed termination, control-channel disconnect or closure, rejected update, or local state corruption requires a new watch snapshot. The client sends UnwatchState when possible, creates a replacement watch, and replaces its cached matching keys only after installing the new snapshot. There is no resume token that lets a client invent missing state. Broad snapshots are bounded by entry count and byte size; a rejected broad watch must be narrowed by namespace or key prefix.

UnwatchState removes a registered watch. A watch update is a notification the client acknowledges with WatchStateAck naming the highest contiguously applied watch_sequence; acknowledgement confirms receipt by the client, not successful external work. The server terminates a watch that overflows its bounded update buffer, misses its acknowledgement deadline, loses authorization, or fails snapshot delivery, and reports the reason in StateStoreWatchClosed.

Media intent

keeppeek.media-intent.v1 is the standard orchestration schema. It represents an intent, not a binding. The document always names stable source IDs and logical stream IDs, never transient source-session IDs or RTP MIDs.

{
  "role": "publish",
  "desired": true,
  "source_id": "front-door",
  "media_kind": "video",
  "variant_id": "browser-h264-720p",
  "output_profile": "browser-h264-720p",
  "priority": 100,
  "recording_mode": "disabled"
}

The required fields are role, desired, source_id, and media_kind. role is publish or subscribe. A publish intent must include recording_mode (inherit, disabled, or required) and the worker copies it to StartPublication.recording_mode; source publication capability and server policy decide whether that request succeeds. A missing or unknown publish-intent mode is a worker validation failure; it must not produce a zero-value StartPublication, which KeepPeek rejects with PUBLICATION_ERROR_CODE_RECORDING_MODE_INVALID. A subscription intent may name an exact variant or leave it absent to permit ordinary automatic selection. The schema can include service-specific declarative configuration under a namespaced parameters object, but it never includes credentials or binary payloads.

The worker flow is deliberately two-stage:

1. A coordinator writes an intent under an authorized service or user namespace.
2. A worker watches the intent and current ServerCapabilities.
3. The worker resolves the stable source ID to the current source session and validates the requested stream, variant, codec, publication, and recording capabilities.
4. The worker sends ordinary Subscribe, StartPublication, StopPublication, or Unsubscribe commands.
5. KeepPeek advertises actual variants and returns actual bindings through capabilities and command results.
6. The worker may write a separate observed-status entry, but it never overwrites desired intent owned by another principal.

This separation prevents stale desired state from granting media access or causing a client to subscribe to a nonexistent stream. A state document saying “publish browser H.264â€� does not mean the variant exists; only a ready MediaVariantCapability does.

Service coordination

A state-store lease prevents duplicate worker ownership without embedding a scheduler into media protocols. For example, workers attempt a create-only key such as service/transcoder-a/leases/front-door-video-browser-h264-720p with a short TTL. The winning worker publishes or renews the lease with compare-and-set. Other workers observe it and stay standby. When the lease expires, a standby can safely attempt a new create-only claim.

The lease determines which service should act. It does not reserve a media variant. The eventual StartPublication remains the authoritative variant ownership check and can still return a conflict if topology changed.

Computer-vision coordination follows the same pattern: intent assigns a model profile to a stable source and stream; the worker starts ordinary subscriptions, publishes only validated events, and reports health separately. Event truth remains in the event catalog and router, not in state entries.

Group coordination

Groups use their own authoritative control state. A client may store a preferred group, local mute layout, or selected audio device in its private state namespace. It must not use shared state to claim membership, grant permission, or assert that another participant is speaking. Those facts come only from GroupState after a successful join.

This distinction prevents an expired preference or a malicious shared-state write from becoming a media authorization decision.

Failure and recovery

• A writer that loses compare-and-set retries after rereading current state.
• A TTL expiry means the owner stopped refreshing; consumers stop acting on that intent unless a newer entry replaces it.
• A watcher reconnects by requesting a fresh snapshot and reconciling desired state with current capabilities.
• A service failure can leave desired state intact while its observed lease expires, allowing a standby to take over.
• An unavailable source session never invalidates stable intent; workers wait for or resolve a new source session through ServerCapabilities.
• KeepPeek restart restores durable state entries according to their TTL and namespace policy, while clients and services reestablish watches.

Backup lifecycle and recovery

The configuration ZIP and the runtime store are two separate recovery sets. A format-3 configuration ZIP contains exactly config.toml and plaintext secrets.toml, with the manifest in the ZIP comment. It contains no database or media. Settings-backed state entries live in the state_store section of config.toml, so they travel inside the configuration ZIP and restore with it. Everything else in the state store needs its own archive.

The generic runtime store is a separate consistent archive and recovery set. The server keeps it in state-store.db inside the configured long-term storage directory and serves reads and watches from memory while committing every mutation to that file first. Back it up as one unit: the state database file plus any accompanying sidecar files, copied while KeepPeek is stopped, or through a tested snapshot procedure that covers every involved volume and writer. A live copy of the database file alone can disagree with its journal or sidecars, exactly as a live catalog copy can. The contract tests in tests/state_store_contract.rs pin the bundle half of this boundary; expiry, quota accounting, and reopen behavior are pinned by the DurableStore unit tests.

Generic dispatch and the keeppeek.state-store.v1 capability turn on only after the database opens and its namespaces, revisions, and entries load into memory. Startup first rejects restored data with excessive counts or bytes, invalid identifiers or schemas, orphan entries, and inconsistent revisions. It checks metadata before decoding bounded documents. If the file cannot open, the server keeps running with an empty in-memory registry, generic commands stay rejected, and the capability stays unadvertised: no client ever sees durability the server cannot deliver.

Every mutation commits to the database before it applies in memory, and the database is the sole revision authority: watch snapshots expire due leases in the database first and replay the exact committed revisions, so disk and cache never report different revisions for the same key. A watch snapshot is additionally bounded to 64 entries and a 64 KiB encoded response; broader watches are rejected so the client can narrow the prefix. When the database is attached, a full store fails the mutation instead of silently evicting the oldest entries.

Restore reinstates counters before watches. A runtime-store restore must bring back namespace revisions, entry revisions, and byte counters together before the server accepts new watches. Clients re-establish watches with a fresh snapshot after a restore and reconcile against current capabilities. No resume token crosses a restore boundary: a sequence gap, a closed watch, or a restored database always means a new snapshot, following the rules in "Watches without a snapshot gap".

Quota, TTL, and full-disk behavior stay explicit. A TTL below one second is rejected; a TTL above 24 hours is clamped to 24 hours. Writes that would exceed the per-namespace entry bound or the total value-byte ceiling are rejected before admission, so a failed write never reserves store space. A storage or disk failure surfaces as a storage error on the mutating call and never reports a silent success, while previously committed entries stay readable. Settings-backed entries reject every TTL because configuration is not a lease.

Measured performance

The reproducible qualification commands are:

cargo test --locked --release -p keeppeek --lib qualification -- --nocapture --test-threads=1
cargo test --locked --release -p keeppeek --lib full_watch_capacity_fanout -- --nocapture --test-threads=1
cargo test --locked -p keeppeek --lib continuous_churn -- --nocapture --test-threads=1

On Windows 11 Pro, AMD Ryzen 5 5600G, rustc 1.98.1, the release workload uses 127 simulated owner IDs across 30 serial CAS rounds (3,810 operations per variant). Both variants construct the same 2,193-byte schema-valid document. The deterministic loop has no random workload seed. Measurements are in microseconds per operation:

Operation	p50	p95	Maximum
In-memory CAS baseline	7	11	119
Durable CAS	853	1,387	3,286
One-entry snapshot	2	3	17

Durability adds 846 microseconds at p50 and 1,376 microseconds at p95. These are control-plane measurements, not media throughput or 127 simultaneous browser sessions. The test rejects any measured operation taking five seconds; that stall ceiling is not a latency service-level guarantee. A 300 ms idle sample measured 0% process CPU and 15,552,512 resident bytes. Sampling after each of 30 rounds observed a maximum of 19,861,504 resident bytes; this is a sampled process maximum, not an operating-system lifetime high-water mark. Run the workload alone or with --test-threads=1 to avoid including another test's work in process measurements.

At full capacity, 1,024 watches across 127 sessions received 30 fanout rounds. Fanout measured 742 microseconds at p50, 860 at p95, and 940 maximum. The workload held 30 pending updates per watch, then verified termination when additional unacknowledged updates exceeded the 32-update bound. This isolates the watch registry with a successful enqueue callback; the separate browser tests cover actual transport delivery and timeout closure.

The quota workload admits 30,601 documents totaling 67,107,993 value bytes, rejects the next document at the 67,108,864-byte ceiling without mutating retained state, reopens the store, and verifies that deletion permits a new write. Namespace and per-namespace entry limits are tested separately.

Continuous disk churn fills and drains 256 entries for 24 cycles on one open store. The observed database-plus-WAL maximum was 4,126,856 bytes; automatic checkpointing reclaimed the WAL without a restart. The regression ceiling is 5 MiB for this fixed workload, not the ceiling for a fully populated store. Opening the store also truncates the WAL. The existing restart-churn test covers 1,024-entry cycles and page reuse after reopening.

Acceptance scenarios

The implementation is complete when these behaviors pass end to end:

1. Two writers cannot overwrite the same orchestration key when compare-and-set revisions differ.
2. A lease expiry reaches every active watcher as one ordered EXPIRE update.
3. A watch started during a write receives either the entry in its snapshot or one later update, never neither or both at the same revision.
4. A transcoder reacts to a publication intent only after resolving current capabilities and successfully owns the actual output variant.
5. A viewer uses a subscription intent to choose a desired variant but waits for its ready capability before subscribing.
6. A group client can persist a preferred group without gaining membership or the right to publish.
7. State values exceeding configured bounds, containing disallowed schemas, or written outside namespace authorization are rejected.