API v3alpha: build stored-mutation watch infrastructure

[siavashs]

Parent: #5450

Summary

Build a transport-neutral, in-process stored-mutation substrate for AlertService and SilenceService watch RPCs.

The broker must be bounded and non-blocking so a slow client cannot stall alert ingestion, dispatch, silence storage, HA merge, configuration reload, or other API traffic.

Mutation broker

Provide:

• Atomic snapshot-plus-subscribe handoff.
• An explicit snapshot-complete event.
• Monotonic node-local sequence numbers.
• Immutable before/after payloads and resource IDs.
• Bounded, non-blocking per-subscriber queues.
• Explicit ResourceExhausted and resnapshot behavior on overflow.
• Cancellation cleanup, stream count, lifetime, and idle controls.
• Metrics for subscriptions, active consumers, overflow, and forced closure.

Cursors are node-local and non-durable. They do not promise replay after restart, failover, or reconnection to another HA peer.

Domain mutation feeds

Alerts

Extend the memory alert provider with non-blocking publication for accepted create, update, delete, and GC mutations. Do not attach one existing provider subscription per RPC client: current listeners can block alert ingestion while the provider mutex is held.

Alert groups

Add dispatcher/group and group-marker observers for stored group creation, update, deletion, content, and marker mutations. Publish only after releasing hot-path locks.

Do not synthesize mutations solely because silence/inhibition evaluation, route configuration, wall-clock time, or time intervals changed computed output.

Silences

Publish local Set/Expire mutations, replacement expire-plus-create, accepted HA Merge changes, and GC purge. Natural pending-to-active and active-to-expired transitions are derived from timestamps and are not mutation events.

Filtering

Derive entered, updated, and exited transitions from immutable before/after data as resources cross subscription filters. Avoid unbounded per-client membership state.

Acceptance criteria

• Snapshot and subscription handoff cannot lose an accepted mutation.
• Every snapshot ends with an explicit snapshot-complete marker.
• Slow subscribers cannot block domain hot paths.
• Queue overflow terminates with documented resnapshot semantics.
• Cancellation and forced closure release queues and admission capacity.
• Node-local cursor scope and restart/HA limitations are documented.
• Alert, group, marker, local silence, HA silence, replacement, and GC mutations are covered.
• Timer-only and recomputed transitions are not emitted.
• Ginkgo/Gomega race and backpressure specs cover ordering, overflow, cancellation, HA merge, and lock isolation.

Dependencies

• Shared contracts and filter semantics: API v3alpha: add shared contracts and reusable infrastructure #5479.
• Stream admission, lifetime, and observability policy: API v3alpha: harden the ConnectRPC foundation #5478.

[siavashs]

The browser watch consumer is tracked in #5492 under the Mantine UI roadmap #5451. It will consume initial snapshots and snapshot-complete markers, reconcile entered/updated/exited mutations, and always establish a fresh snapshot after queue overflow or loss of node-local cursor continuity. The UI will not imply durable or cluster-wide replay.