Release v19.109.0

This release makes cluster startup, replay and SQL storage safer and faster. Readiness and liveness are now reported separately, and replay hands each page of events to a projection in one call. The release also adds MongoDB writer fencing for projection generations, SQL schema operations for least-privilege deployments, and native upserts.

Added

• With Features.ProjectionGenerations on, every MongoDB read-model sink write runs in one majority transaction that checks the writer's fence gate, applies the writes and records a receipt. A revoked writer's batch fails with FenceRevoked, and a commit whose outcome is unknown is resolved before the batch is retried or reported. (#5225) (part of #5159)
• Cratis:Chronicle:ReadModels:MaxBatchSize (default 500) bounds how many documents one fenced sink write covers. Larger bulk flushes are split into several fenced transactions. The value must be positive. (part of #5159)
• The kernel refuses to start with projection generations on when MongoDB runs as a standalone server, because the fence needs transactions. The message explains how to run MongoDB as a replica set with --replSet. (part of #5159)
• /health/live reports healthy as soon as the kernel process is listening, without running any health checks, for liveness probes. See Health endpoint. (part of #4040)
• /health/ready reports healthy only once the silo has started and the kernel can serve requests, for readiness probes and load balancers. (part of #4040)
• --migrate-only server option: applies the SQL schema migrations for the cluster, every known event store and namespace, and the jobs schema, then exits without starting the server. Also available as Cratis__Chronicle__Storage__SchemaAction=MigrateAndExit. (#5204)
• --verify-schema server option: checks that the SQL schema is current without changing it, then exits. Also available as Cratis__Chronicle__Storage__SchemaAction=VerifyAndExit. (#5204)
• Repeatable --event-store <name> and --namespace <name> options with --migrate-only provision a new event store and namespace ahead of time, so a server without DDL rights can use them. (#5204)
• Cratis__Chronicle__Storage__SchemaMode=Verify (Cratis:Chronicle:Storage:SchemaMode): the server never creates databases or schemas and never applies migrations. It fails at startup, or when an unprovisioned event store or namespace is first used, with an error that says to run --migrate-only. The default, Migrate, keeps today's behavior. (#5204)
• Schema actions exit with 0 (migrated or current), 2 (schema not current, including a missing database during verification), 3 (schema lock timed out), 4 (invalid options, or a database that CreateIfMissing=false forbids creating) or 1 (other errors). On MongoDB and in-memory storage they succeed without doing anything. (#5204)
• New guide for least-privilege SQL deployments: SQL schema operations. (#5204)
• SQL storage now reports schema-lock wait time and recovered insert collisions as metrics, logs which node applies each migration, and the health endpoint reports unhealthy until startup has completed. (#5206)
• PostgreSQL storage can authenticate with Microsoft Entra (managed identity) natively through Cratis:Chronicle:Storage:PostgreSqlEntra, acquiring and refreshing tokens automatically instead of passing a static access token as the password. (#5207)
• EventStoreForTesting.ShareApplicationMetadata (on by default) lets new test event stores over the same artifacts reuse discovered event types, model-bound projection definitions and seeded event type definitions. Each store still gets its own copy of every mutable object. Set it to false to discover per store; custom service providers, explicit artifacts and ShareSchemaGenerator = false always discover per store. (#5279)

Changed

• While the silo is still starting, gRPC calls to the kernel fail with Unavailable and HTTP requests, including the Workbench, return 503 Service Unavailable, so clients retry instead of hanging. Previously the ports were not open at all until startup finished. (part of #4040)
• /health keeps its meaning and is the same as /health/ready. Point liveness probes at /health/live: a liveness probe on /health restarts a silo that is still working through its replay backlog. (part of #4040)
• On a dedicated, exclusive health port, /health/live and /health/ready are served next to the configured health endpoint. (part of #4040)
• Projection replay now hands each page of events to the projection in one call, in global sequence order, instead of one call per run of events from the same event source. Replays over interleaved event sources make about half as many grain calls, and a failing event still fails only its own partition at the right sequence number. (#5262)
• SQL storage (SQLite, PostgreSQL, SQL Server) now writes through the database's native upsert, so concurrent writers no longer collide on the same key and no longer need retry-after-conflict recovery. (#5201)
• Creating an EventScenario, a ReadModelScenario or a test event store with new Defaults(...) takes a fraction of the time and memory it did in applications with many event types and read models. The first store in a process still does the full discovery. (#5279)

Fixed

• A client-hosted (gRPC) reactor or reducer partition whose delivery timed out is held until the original client finishes or the hold expires, instead of being delivered to a second client while the first may still be handling it. (#5014)
• A partition held after a timed-out delivery stays held when its failures are cleared and the observer restarts, so it is not delivered to a second client while the first may still be handling it. (#5242)
• A deferred projection child is still resolved when the parent's existing children are keyed by a different type than the lookup key, instead of the deferred child never being applied. (#5247)
• Activating an observer, webhooks or event store subscriptions no longer waits behind long-running catch-up or replay work on the same grain, so a silo joining a cluster with a replay backlog finishes starting instead of timing out. (part of #4040) (#3852)
• The projection editor help shows the $eventContext(Property) syntax and the event context properties that actually exist, instead of $eventContext. with a causationId property that is not there. (#4117)
• Observers no longer skip unhandled events when they are re-activated after their last handled event moved ahead of the next event to handle. (#5304)
• Observers that subscribe to all events now catch up correctly when using SQL storage. (#5305)
• Several Chronicle kernel instances starting at the same time against one MongoDB database no longer fail with E11000 duplicate key errors. Startup writes for system information, cluster state, event store initialization, encryption keys and the security stores now accept a document another instance already wrote, while a conflicting document is still reported as an error. (#5203)
• In a rolling upgrade, only the newest kernel applies patches: an older node no longer runs or answers patch application after a newer kernel has patched the store. (#5205)
• Replaying a reactor or other observer that reads from an event sequence other than the event log, such as an inbox, now replays it instead of reporting success without doing anything and leaving the observer Disconnected. Observer commands that omit the event sequence now use the observer's own sequence. (#3701)
• The Workbench now tells you why a replay could not be started. (#3701)
• Testing scenarios now refuse a public event type appended to the event log and a private one appended to the outbox, as a deployed kernel does, instead of passing in tests and failing at runtime. (#5254)
• Primary read model lookups during a replay on MongoDB no longer query the collection list on every hit, and a lookup that meets a collection promotion no longer logs a warning unless every retry fails. (#5258)
• SQL storage applies migrations in release order on a fresh database, so a migration from a 19.100+ release no longer runs before the 19.10–19.99 migrations it builds on. (#5244)
• SQL storage no longer fails or duplicates rows when several kernel nodes cold-start against the same database at once. (#5202)