Stratara 4.4.7

A release for hosts on the Orleans execution model under load. A burst of heavy commands no longer costs
attempts it never ran. A long heavy or runner unit no longer leaves timeout warnings in the logs. A host that
commits facts without hosting a reader can wake the store readers instead of needing a message broker. A host can
wait for exactly its own write. The PostgreSQL registrations keep the pool size a connection string names. The
dependencies move to Orleans 10.4 and StackExchange.Redis 3.4. Additive API only; no schema change and no migration.
Two behaviours change on purpose, see Upgrade notes.

Added

• A mark of what a host just wrote: IProjectionProgress.MarkAsync(IEnumerable<Guid> streamIds). Taken after a
  commit, it covers every entry committed to those streams and holds up WaitAsync only in the partitions they fall
  into. A host reading its own write no longer waits for the projection to catch up with the whole store, and a
  partition that lags or is stopped elsewhere does not hold it up. MarkAsync() stays the mark for decisions that
  need everything committed. The default interface implementation answers the whole-store mark, so an implementation
  of your own keeps compiling and keeps the promise.
• ICommandIntentStore.RenewManyAsync (default: one RenewAsync per command) and
  HeavyWorkOptions.QueuedLimitPerPool (default 256). See Fixed for what they serve.
• AddStrataraStoreWakeUps() (Stratara.Orleans). A host that commits facts on the Orleans execution model
  without hosting a reader can now wake the store readers instead of publishing bundles to the bus. Such a host is an
  Orleans client that runs command handlers in process, or a command silo without a store-reading role. After a
  commit, it wakes the readers of every projection and saga role in the cluster for the partitions the commit
  touched, through the host's grain factory, and stores nothing. With it, such a host needs no message broker. Until
  now its commits failed against the missing broker and stayed in the outbox table. The host needs to know no
  projection or saga. A silo of each role wakes its own readers, whatever partition count it runs with. On a silo
  that hosts one store-reading role, the other role is woken through the cluster as well. A role no silo hosts is
  set aside for StoreWakeUpOptions.AbsentRoleRetry (ten seconds) instead of being called on every commit, with
  117_131 when it is found absent and 117_132 when it is woken again. At most 256 wake-ups of a role await an
  answer at once, so a silo that hangs cannot pile up calls on the committing host. hybrid: true keeps publishing to the bus
  during a rollout, and the registration combines with AddStrataraProjectionGrains and AddStrataraSagaGrains in
  either order.

Changed

• Dependencies. The packages now depend on Microsoft.Orleans.* 10.4.0 (from 10.3.1) and
  StackExchange.Redis 3.4.5 (from 3.1.31), along with the current patch releases of Microsoft.Extensions.*,
  ASP.NET Core and EF Core 10.0.13, and OpenTelemetry 1.20.1. A host that pins Orleans below 10.4 has to move with
  them. StackExchange.Redis 3.4 reworks the buffer of its connection writer, in which 3.1 could crash the process
  with an access violation while replaying its backlog after a reconnect. The Orleans Redis grain directory hit that
  path in the framework's own tests.
• Migrate to the Orleans execution model no longer says an API host commits nothing. Its broker table now
  separates a host that only dispatches from one that runs handlers in process, and names the registration that
  removes the broker from each.

Fixed

• Waiting no longer costs a recorded command its attempts. Under a burst of heavy commands many times the
  cluster-wide limit, every accepted command held a lease that renewed its record with a statement of its own. The
  store load grew with the queue until the connection pool was exhausted. Renewals then failed, records lapsed, the
  drain handed them over again, and each hand-over counted an attempt, so commands were kept for an operator although
  no handler had failed. Now:

  • a silo renews everything it holds in batches, one statement per thousand commands per renewal period;
  • one heavy pool holds at most QueuedLimitPerPool commands, and a hand-over beyond it is refused, so the burst
    waits in the store;
  • a resumed command's attempt is counted when a silo takes its hand-over over, in
    ICommandIntentStore.TryRenewFromAsync, not when the drain claims it. A refused hand-over, or one that finds its
    command still held, counts nothing. A first hand-over the pool refuses gives back the attempt its dispatch
    counted.

  A stop already gave back the attempts of what it ended. A handler that runs and fails still counts, and is kept at
  the bound as before. The renewal loop runs until the silo has stopped, not until the host does, so a handler still
  running during a stop keeps its record renewed. New log events: 117_133 (HeavyPoolFull, Debug), 117_134
  (IntentBatchRenewalFailed, Warning) and 117_135 (AttemptGiveBackFailed, Warning).

• A pool size the connection string names is kept. The PostgreSQL registrations of the write, read and identity
  contexts rewrote every connection string whose maximum pool size was Npgsql's default of 100 to 15, including one
  that named Maximum Pool Size=100 on purpose, and said nothing about it. They now apply 15 only where the
  connection string names no pool size, and log the size in effect once per host and database (102_008, category
  Stratara.EventSourcing.EntityFrameworkCore.ConnectionPool, with the database and the host, never the password). Getting started — DI composition explains the default and that
  contexts with one connection string share one pool.

• A hand-over is answered once the command is accepted. The hand-over of a heavy command, and of a command
  that names no aggregate, stayed open until the command had run. A unit longer than the runtime's response
  timeout made Orleans log Response did not arrive on time on the dispatching host and several Dropping expired
  message warnings on the silo, even though the unit ran to its end and committed. Both are now answered once the
  silo holds the command and its lease has started, as the hand-over of an aggregate's command already was, and the
  command runs on the silo however long it takes. Lease, permit, completion, failure recording and resumption are
  unchanged. A hand-over that times out now means the silo did not accept it in time, and it is logged as a failed
  hand-over like any other failure.

• Stopping a silo no longer waits out the host's shutdown budget behind a running heavy unit. The heavy-work
  runner waited for its running units when the host stopped it. The host stops it before the silo, and a unit is
  cancelled only once the silo stops, so the host's stop hung until its shutdown timeout and the unit was not
  cancelled within the deactivation budget. The runner now stops taking units and leaves the running ones to the
  pool's deactivation, which waits for them within the budget and then cancels them, so they are resumed elsewhere.

• A cancellation the silo's stop did not cause is a failed attempt. A recorded command whose handler threw an
  OperationCanceledException of its own, for example from an HTTP call that timed out, was neither recorded as a
  failure nor logged. It now counts against the delivery bound like any other failure, with IntentAttemptFailed
  and its message on the record. A handler cancelled because its silo stops still gives its attempt back.

• A duplicate hand-over to a command that names no aggregate is logged. The runner now holds the command from
  the first hand-over on. A second hand-over that arrives meanwhile is dropped with IntentHandOverDropped at Debug,
  as one dropped by the fence already was. A hand-over that reaches the runner while its silo stops is given up at
  once, with its attempt given back, as the aggregate grain already did.

• Documentation. Operate the Orleans execution model no longer says that a long hand-over's timeout goes
  unlogged: only a forwarded command is bounded by the response timeout. Migrate to the Orleans execution model
  no longer says that Orleans' PostgreSQL scripts ship inside the ADO.NET provider packages. They do not, and the
  guide now lists the paths to take them from in the dotnet/orleans repository.

Upgrade notes

• A connection string that names Maximum Pool Size=100 now gets 100. Before, the PostgreSQL registrations
  rewrote it to 15 as if it named nothing. A host that named 100 and relied on getting 15 should name 15.
• ICommandIntentStore implementations of your own. The framework's store now counts a resumed attempt in
  TryRenewFromAsync instead of in TryClaimAsync/ClaimAsync. A store that keeps counting at the claim still
  bounds resumption, but counts waiting as well; move the increment to keep waiting free. Implement RenewManyAsync
  for one statement per batch. QueuedLimitPerPool below what a pool runs at once fails the start.
• Upgrade the silos before the hosts that dispatch. A dispatcher of this release that hands over to a silo of an
  earlier one still waits for the whole unit there. Because it no longer keeps that wait's timeout quiet, it logs a
  failed hand-over (117_113 HandOverFailed, Warning) for every long unit until the silo is upgraded, beside the
  runtime's own warnings. The unit itself runs and completes, and the drain does not hand it over again.
• Upgrade the silos before registering the wake-ups on other hosts. A silo of an earlier release has no wake-up
  grain, so a wake-up placed there is lost, logged at debug, and its readers find the commit by their poll.