Release v16.8.0

Summary

Performance and correctness pass over the kernel's append, observer, and projection paths.

Added

• Benchmarks for projection, reducer and reactor throughput, projection replay, and passive-versus-materialized read models.
• Benchmarks comparing a single silo against a two-silo cluster, including a split-role topology.
• Startup warning when a clustered deployment is configured against local-only storage.
• Clustering configuration documentation covering environment requirements, multi-node compose, and placement behavior.
• Jobs.StepCheckpointFlushInterval, bounding how long a job step may hold an unpersisted checkpoint.
• ReadModels.GuardSinkWritesOnWatermark, the kill switch for the crash-recovery write guard.
• Events.ConstraintsVersionCheckInterval, bounding how often an active event sequence checks for constraint changes.

Changed

• Reduced storage round-trips and CPU on the append, observer, and projection paths through event-type schema caching, debounced observer-state persistence, memoized projection key resolution, striped projection locking, subscribed-queue routing, and coalesced observer dispatch.
• Appends no longer block on slow observer consumption; a full delivery queue spills to the observer catch-up path instead of applying backpressure.
• Event batches are routed only to queues whose subscribed observers care about the batch's event types, and queues prefer co-location with their event sequence.
• Projections whose key collapses several partitions onto one read-model document now handle all partitions through a single activation.
• Documentation no longer claims projections and reducers are idempotent across a crash; that guarantee covers a deliberate replay only.
• SaveChanges, ProjectionPipelineManager and ReducerPipelineFactory each take an additional constructor parameter, which is source-breaking for anyone constructing them directly.

Fixed

• Projection read models could be corrupted in a multi-silo cluster when replay and catch-up ran the read-modify-write cycle concurrently for a partition-collapsing key.
• Accumulating projections and reducers no longer double-count events re-delivered by crash recovery, except where a resumed catch-up re-cuts the original batch boundaries.
• Every replay job reported CompletedWithFailures, because benign step-start outcomes were counted as failures and no failed-step record was written.
• Resuming a job reported a healthy running step as a failed resume, and reported genuine resume failures as successes.
• [Unique] could reject a value nothing held on in-memory storage, because changing a constrained value never released the previous one.
• Joined projection changes after the first were silently dropped, leaving a stale read model with no error.
• Concurrent resolution of event-store and namespace storage could serve a stale event-type schema or leave two sinks for one read model with divergent replay state.
• A cached event-type schema was never invalidated when the event type changed, so properties present on the wire but missing from the stale schema were silently dropped between silos.
• A job step that completed after it was no longer tracked turned a successful completion into an opaque job error.
• Job lookups by type and status always returned nothing against MongoDB, so running and stopped jobs were invisible and were restarted rather than resumed.
• An observer could stop processing events permanently when preparing a catch-up never produced a job.
• An exception while entering or leaving a state left an observer unable to transition again.
• A durable append could be reported as failed after the event was persisted, inviting a retry that appended it twice and skipping the unique-constraint index.
• Queue assignment used a per-process randomized hash, so a restart could re-map observers to different queues.
• Grain timers scheduled for replay-on-definition-change were retained for the lifetime of the activation.
• Constraint index resolution during AppendMany stamped each event's index with the wrong sequence number.
• Constraints registered after an event sequence or observer had activated were not enforced.
• Server crashed on startup against MongoDB due to unsupported LINQ in job index creation.
• Concurrent changes to the in-memory read-model sink could race.
• Duplicate-key failures when writing a unique-constraint index surfaced as a raw storage exception.

Removed

• IEventTypesStorage.Populate(), obsolete now that schemas are read through the cache.
• ObserverState.HandledEventCountPerPartition, replaced by a dedicated store.
• IServer.ReloadState() and its host.proto RPC, non-functional since inception.

Security

• Stop logging raw Authorization header values.