Release v19.110.0

Added

• Localhost clustering with persistent storage now uses a database-backed single-writer lease to prevent overlapping kernels from processing the same storage. It is enabled by default and configurable under Clustering:SingleWriterLease; a kernel stops if it cannot renew its lease for half the lease time. MongoDB requires version 5.0 or later. Start-before-stop deployments can time out safely while the old instance holds the lease, and a development kernel-state reset keeps the lease of the running kernel. (#5109)
• Quarantined webhooks now recover automatically by periodically retrying the oldest failed event, honoring receiver deferral deadlines. A successful probe lifts quarantine and refreshes partition retry budgets; failed probes do not consume them. Configure this under Webhooks:Probe. Quarantine survives kernel restarts; partitions parked after 410 Gone or by earlier versions still require manual retry. (#5108)
• An explicit, idempotent per-event-store generation initializer records durable receipts and blockers and refuses adoption when it cannot prove exclusion. It is not run automatically and does nothing while projection generations are disabled. (part of #5223)
• The kernel records the projection definition version each connected client registered, providing evidence for deployment-gated promotion when Features.ProjectionGenerations is enabled. (part of #5161)
• Webhooks can authenticate with the kernel's Azure managed identity or workload identity instead of a stored client secret: register with WithManagedIdentity(scope, clientId) in the .NET client or choose "Managed identity" in the Workbench. See https://github.com/Cratis/Chronicle/blob/main/Documentation/hosting/azure.md (#5208)
• Read models can declare composite indexes with [CompositeIndex] and descending indexes with [Index(IsDescending = true)], on MongoDB and SQL read-model stores. (#5199)
• SQL read-model indexes can target nested properties, and PostgreSQL indexes collection paths with a GIN index; SQL Server and SQLite skip collection indexes with a warning. (#5198)
• IUnitOfWorkCompletionClaim lets a caller that owns the request's unit of work (such as Arc) claim its completion, commit it itself and read the outcome as a UnitOfWorkCompletionResult (Committed, Failed, RolledBack or Unknown) before it writes the response. Unknown means the commit failed technically and events may or may not have been appended. A claimed unit that is never completed is rolled back and logged as an error. (#4864)
• ChronicleAspNetCoreOptions.UnitOfWork.ReportFailuresBeforeResponseStarts returns ProblemDetails for a failed commit when nobody claimed the unit and the response has not started: 409 for conflicts, 400 for constraint violations, 500 otherwise. Off by default. MVC and minimal-API apps that write their response before the commit keep today's behavior (see #5339). (#4864)
• Custom sinks can opt in to atomic isolated replacement and guarded upserts, used by upcoming partition rebuilds. (part of #4334)
• A materialization binding catalog for MongoDB and in-memory storage that records, per read model, which generation is served and under which serving epoch. Publishing a new generation is one conditional, idempotent operation that revokes the previous generation's write fence, and generation numbers are never reused. SQL storage refuses the catalog explicitly. (part of #5159)
• The clustering benchmark can rebuild a projection under scenarios S1-S4, cold or under live load, at small, medium, large or custom sizes and on one, two or three silos. Each run prints a JSON record with rebuild throughput, time to served swap, post-swap catch-up, served-read latency, MongoDB command and grain-call counts, per-silo grain activations and a verdict against a never-rebuilt twin projection, and exits non-zero on a mismatch. (part of #5156)

Changed

• Startup admission discovers generation floors even after a kernel state reset, without creating or registering stores, so kernels older than a store's floor are refused. Kernel state resets are refused for stores whose generations were initialized. These changes do not enable projection generations. (part of #5223)
• Store-wide event type statistics use bounded concurrency, support cancellation and fail rather than return partial totals when a namespace cannot be read. Refused observer replays preserve progress and release the observer; partition replay markers are released when jobs cannot start, and unsupported global replays are rejected before recovery or finalization. (#5220)
• Projection replay notifies read model watchers once per page of events instead of once per changed read model, reducing internal calls and improving replay speed. (#5321)
• The replay-shape benchmark reports sink bulk sizes, checkpoint fan-outs and event cursor pages. (part of #5260)
• A unit of work commit that fails after the request ran is now logged as an error instead of dropped silently. (#4864)
• After a unit of work completes, IUnitOfWorkManager.Current and HasCurrent no longer return it; they return the unit that was current when it began, if still open. (#5102)
• Dropping, renaming or clearing a read model collection that a projection generation owns is refused. (part of #5159)
• The kernel now reports job removal failures instead of treating them as removed.

Fixed

• Resuming a running MongoDB projection catch-up now preserves its open bulk window instead of discarding unflushed writes, which could silently omit read model changes before the first checkpoint or later fail with BulkWindowLost. Kernels 15.0.0 through 19.109.0 are affected; PostgreSQL, SQL Server and in-memory storage are not. Replay MongoDB projections that caught up while clients restarted or re-registered, and those whose logs show BulkWindowLost or catch-ups completed with failures; silent losses leave no log entry. (#5340)
• Observers stored with SQLite, PostgreSQL and SQL Server retain their total and per-event-type handled event counts instead of always reporting zero. (#5299)
• PostgreSQL development kernels release idle pooled connections when kernel state is reset, preventing connection exhaustion during extended use. (#5322)
• Bounded event reads for projections with several read models, including GetInstances with an event count and projection previews, populate initial read model values consistently with running projections. (part of #4594)
• Failed spills of appended-event queues are retried on a short timer rather than waiting for another append. (#5246)
• MongoDB projection catch-ups and replays no longer accept a bulk write window that was replaced before their first checkpoint, and resuming a replay or catch-up no longer carries queued writes into the other collection. This hardens the fix for resumed catch-ups. (#5341)
• [Index] on properties of a nested type that is used more than once in a read model now creates an index for every use. (#5198)
• Completing a nested unit of work restores the unit that was current when it began, so the request unit can still be claimed afterwards. (#5102)
• Replay, catch-up and lagging live processing now resolve a child's parent key as it was when the child event happened, not by the parent's latest key. Read models with content-keyed parents that were rebuilt before this fix may hold children under the wrong parent; replay them to correct it. On startup, SQL event sequence tables (including existing ones) get a new index on event source, type and sequence number, and MongoDB gets a matching eventSourceId_eventTypeId_sequenceNumber index. (part of #5328)

Security

• Managed identity webhooks are off until configured: the kernel refuses them, at registration and at delivery, unless Webhooks:ManagedIdentity:AllowedScopes lists the scope, Webhooks:ManagedIdentity:AllowedClientIds lists the client id (or AllowSystemAssigned is set for the system-assigned identity), and the receiver host is in the webhook host allow-list (Webhooks:AllowedHosts / EventStoreAllowedHosts), even when validation is Warn. The token is never logged or recorded in delivery failures. (#5208)