HostLoom 0.9.0

The ninth release of HostLoom. Five threads run through it. The first is scheduled work. HostLoom.Scheduling is a scheduled-job kernel in the shape of the caching and locking kernels: IScheduledJob is the unit of work, ScheduleTrigger.Cron accepts five or six fields with names, steps, ?, Sunday as 0 or 7, either-day matching, and a time zone with daylight-saving handling, FixedRate counts from the previous due time and FixedDelay from the previous completion, and a Scheduler runs each schedule as one sequential loop over TimeProvider with a per-run timeout, typed outcomes, state, an execution-free probe, and the HostLoom.Scheduling meter and activity source. A late run starts at once and missed occurrences are not replayed; a failed run is logged and the schedule continues. An Exclusive schedule runs on one instance at a time through IScheduleGuard: a claim is granted or refused at once, a refused claim skips the run, a guard failure skips it as GuardFailed rather than running unguarded, and a claim is held past the run until the lease ends or the next occurrence is due, so a slower instance reaching the same occurrence is refused. HostLoom.Scheduling.DependencyInjection registers schedules with a fresh scope per run and validates at startup that an exclusive schedule has exactly one guard, HostLoom.Scheduling.Locking supplies DistributedLockScheduleGuard over IDistributedLock with one skip-if-busy acquisition of schedule:{name} per run, and HostLoom.Scheduling.Testing supplies ManualScheduleGuard, scripted with Hold, Release, Lose, and FailNext.

The second thread is leadership. HostLoom.Leadership is lease-based leader election over the distributed lock: ILeadership exposes Role, IsLeader, Term, LeadershipToken, WaitForLeadershipAsync, and OnChange, and LeaderElector runs a candidate loop on a jittered retry cadence, renews explicitly without being capped by Locking:MaxHold, steps down at once on a refused renewal or an expired lease, resigns on request, and releases on stop. The guarantee is stated as the lock's: at most one leader while clocks and the backend behave, a window bounded by the lease on expiry, and no fencing; Term counts this instance's acquisitions. HostLoom.Leadership.DependencyInjection registers electors keyed by role, HostLoom.Scheduling.Leadership supplies LeaderScheduleGuard and UseLeader(role), which run exclusive schedules on the elected leader with no lock round trip per occurrence, and HostLoom.Leadership.Testing supplies ManualLeadership. LeaderChannel<T> is a bounded Channel<T> gated on a role: every instance writes to it, a leader's items reach the reader, and a follower's are accepted and discarded, so a warm standby keeps its state current without acting; drops are counted by channel and reason and summarised in the log at most once per interval. A leader elector whose lock is disabled no longer leads on every instance: LeadershipOptions.WhenUncoordinated defaults to Follow, and a single-instance or development host opts into Lead. Two-elector tests over the in-process lock and a fake clock, a real-Redis test, and an opt-in outage experiment measure the hand-over window against the lease.

The third thread is reliable messaging at the edges of a unit of work. A transactional outbox routes every IPublishEndpoint.PublishAsync through an IOutboxStore from the caller's scope, and OutboxRelay drains it when woken and every Outbox:PollInterval, publishing each frame unchanged through the transport; a failed message is left pending with its error, retried with exponential backoff, and dead-lettered after OutboxOptions.MaxAttempts. Delivery is at-least-once. An idempotent inbox records {topic}:{subscription}:{messageId} before the handlers run, skips a delivery seen inside the window with an InboxDuplicate payload, runs anyway with an InboxSkipped payload when the store cannot answer, and passes requests through untouched; InboxStore.FromClaim builds a cache-backed store in one line. HostLoomBuilder.ConfigureReceivePipeline accepts a filter that needs a service from the container, and the pipeline is still composed once, from the root provider. RabbitMQ event publication now waits for publisher confirmation and sets persistent delivery when DurableTopics is enabled, so an unconfirmed outbox message remains retryable.

The fourth thread is a security hardening of the whole solution. Remote faults no longer carry the handler's exception type and message by default; a handler that wants the caller to see a message throws RemoteFaultException, and HostLoomOptions.IncludeFaultDetails restores the previous behaviour. The WebSocket gateway refuses a keyless subscribe to a keyed topic unless the registration passes allowTopicWideSubscription: true, drops an oversized event instead of aborting every subscriber, bounds how long a subscriber can hold a snapshot provider open, budgets request frames per second, resolves every configured authorization policy name at startup, and ends a subscription before it answers an invalid credit or ack frame so a fault after subscribed is terminal on both peers. Kafka validates the reply-to header before the handler runs and gains SecurityProtocol, SaslMechanism, and TLS settings; RabbitMQ accepts only server-named reply queues by default, declares a dead-letter exchange when configured, and moves to versioned, role-qualified queue names. Envelope decoding rejects empty ids, the messaging.message.type tag is bounded, logging caps message and field lengths and honours [NotLogged] and [LogMasked] on overridden virtual properties, the Redis and Valkey invalidation channels drop malformed messages and reject tag members outside the namespace's data prefix, the single-flight guard map is bounded, CacheKey.FromSensitive and LockKey.FromSensitive hash low-entropy secrets with HMAC-SHA256, and every coordinated contract now reports IsCoordinated.

The fifth thread is a production review of the cache and lock kernels. Lock acquisition rejects a successful backend reply that arrives after the usable lease has expired, concurrent renewals serialize backend calls with local deadline updates so an older renewal cannot overwrite a newer deadline, a heartbeat that fails re-arms itself and keeps retrying until an extension succeeds or the lease runs out, and a release that throws is logged only, because the backend keeps the owner's lease until it expires. RedisConnection reports the client name of a multiplexer supplied by Redis:ConnectionFactory, so tracking finds that connection's subscriber; a tagged Redis write pipelines the tag memberships before the value; the Valkey invalidation channel flushes after a first subscription that follows failed attempts and probes itself so a dead subscriber socket is replaced within one CommandTimeout; and the cache tracks invalidation generations in 1024 key stripes and recognises the echo of its own publish, so an invalidation of one key no longer suppresses every in-flight fill on the instance and the refill that follows a removal is cached rather than dropped. CacheFilter runs the rest of the pipe on a remembered null, the IDistributedCache adapter applies Caching:MaxPayloadBytes, Kafka reply consumers retain progress across reassignments, and RabbitMQ subscriptions close channels even when cancellation callbacks throw.

Upgrading from 0.8.0 is source-breaking for custom implementations: IsCoordinated is added to IDistributedLock, ILockHandle, ILeadership, and IScheduleGuard, IOutboxStore.MarkFailedAsync takes the next attempt time and MarkDeadLetteredAsync is new, and OutboxMessage, WebSocketTopicDescription, LeadershipDescription, and SchedulingDescription gain positional record parameters; every in-repo implementation and the Testing packages are updated. Several defaults change: remote fault details are opt-in through HostLoomOptions.IncludeFaultDetails; RabbitMQ deployments must follow the queue migration guide or set RabbitMqOptions.QueueNaming to RabbitMqQueueNaming.Legacy during rollout, and a client-named reply queue needs RabbitMqOptions.AllowNamedReplyQueues; a keyed WebSocket topic refuses a topic-wide subscription unless the registration allows it; a leader elector with the lock disabled follows unless LeadershipOptions.WhenUncoordinated is Lead; the Kafka reply consumer starts from the latest offset; the inbox key is length-prefixed, so keys recorded before the upgrade are not recognised for the remainder of their window; and a restricted Valkey cache ACL now also needs SREM. Each is stated under Changed, Fixed, and Security below.

Eight packages are published for the first time: HostLoom.Scheduling, HostLoom.Scheduling.DependencyInjection, HostLoom.Scheduling.Locking, HostLoom.Scheduling.Leadership, HostLoom.Scheduling.Testing, HostLoom.Leadership, HostLoom.Leadership.DependencyInjection, and HostLoom.Leadership.Testing. HostLoom.Scheduling and HostLoom.Leadership depend only on the logging abstractions, with HostLoom.Leadership and HostLoom.Scheduling.Locking building on HostLoom.Locking; the DependencyInjection packages add the hosting and options abstractions, and the Testing packages depend on their kernel alone. @hostloom/websocket-client is not part of this NuGet set; that package carries its own version and ships when a websocket-client-vX.Y.Z tag is published.

Added

• HostLoom.Scheduling, a scheduled-job kernel in the shape of the caching and locking kernels: IScheduledJob, ScheduleTrigger.Cron (five or six fields, names, steps, ?, Sunday as 0 or 7, either-day matching, time zones with daylight-saving handling), FixedRate counted from the previous due time, FixedDelay counted from the previous completion, and a Scheduler that runs each schedule as one sequential loop over TimeProvider with per-run timeout, typed outcomes, state, an execution-free probe, and the HostLoom.Scheduling meter and activity source. A late run starts at once and missed occurrences are not replayed; a failed run is logged and the schedule continues. An exclusive schedule keeps its claim past the run until the lease ends or its next occurrence is due, so a slower instance reaching the same occurrence is refused; ScheduleState.ClaimHeldUntil reports the hold. Two-instance cluster tests cover local and exclusive schedules, hand-over on stop, a lock backend outage, and a lost lease, and a real-Redis contention test plus an opt-in outage experiment run in the integration suite.
• IScheduleGuard, the contract that runs an Exclusive schedule on one instance at a time: a claim is granted or refused at once, a refused claim skips the run, and a guard failure skips it as GuardFailed rather than running unguarded.
• HostLoom.Scheduling.DependencyInjection: AddHostLoomScheduling, AddSchedule<TJob> with a fresh scope per run, a delegate overload, UseGuard<TGuard>(name) with the exactly-one rule, startup validation naming the builder method an exclusive schedule needs, and the hosted service that starts and stops the scheduler with the host.
• HostLoom.Scheduling.Locking: DistributedLockScheduleGuard over IDistributedLock, one skip-if-busy acquisition of schedule:{name} per run with automatic extension up to Locking:MaxHold, and UseDistributedLock().
• HostLoom.Scheduling.Testing: ManualScheduleGuard, scripted with Hold, Release, Lose, and FailNext, recording every claim.
• HostLoom.Leadership, lease-based leader election over the distributed lock: ILeadership (Role, IsLeader, Term, LeadershipToken, WaitForLeadershipAsync, OnChange), LeaderElector with a candidate loop on a jittered retry cadence, explicit renewal that is not capped by Locking:MaxHold, immediate step-down on a refused renewal or an expired lease followed by one retry interval, ResignAsync, release on stop, a probe, the HostLoom.Leadership meter and activity source, and log events 3400 to 3408. The guarantee is stated as the lock's: at most one leader while clocks and the backend behave, a window bounded by the lease on expiry, and no fencing; Term counts this instance's acquisitions.
• HostLoom.Leadership.DependencyInjection: AddHostLoomLeadership().AddRole(role, configure), electors and ILeadership keyed by role with the single role also resolving unkeyed, named options validated per role, and the hosted service that starts and stops every elector.
• HostLoom.Scheduling.Leadership: LeaderScheduleGuard and UseLeader(role), which run exclusive schedules on the elected leader with no lock round trip per occurrence; a run in progress ends as ClaimLost when leadership ends.
• HostLoom.Leadership.Testing: ManualLeadership, flipped with Acquire, Lose, and Resign.
• LeaderChannel<T> in HostLoom.Leadership, a bounded Channel<T> gated on a role: every instance writes to it, a leader's items reach the reader, and a follower's are accepted and discarded so a warm standby keeps its state current without acting. LeaderChannelOptions sets the capacity, full mode (DropOldest by default), and drop report interval, validated by key. Drops are counted on hostloom.leader.channel.dropped by channel and reason (follower or full) and summarised in the log at most once per interval, with the tail flushed when the instance becomes leader and on disposal: LeaderChannelFollowerDropped (3409) at Information and LeaderChannelFull (3410) at Warning. Fault tests feed a channel on each of two electors through a refused renewal, a provider outage, racing writers, a throwing listener, and disposal mid-term, and a real-Redis case with the leader cut off measures the items that land on both instances against the lease.
• Two-elector tests over the in-process lock and a fake clock, and a real-Redis test plus an opt-in outage experiment that measures the hand-over window against the lease.
• A transactional outbox in HostLoom: IOutboxStore (append inside the caller's unit of work, atomic claim with a lease, mark published or failed), OutboxMessage carrying the encoded frame, UseOutbox<TStore>() and UseInMemoryOutbox() on the builder, which route every IPublishEndpoint.PublishAsync through the store from the caller's scope, and OutboxRelay, a hosted loop that drains when woken and every Outbox:PollInterval, publishes each frame unchanged through the transport, and leaves a failed message pending with its error. Delivery is at-least-once. Metrics hostloom.outbox.published, hostloom.outbox.failed, and hostloom.outbox.lag; log events 3300 to 3303.
• An idempotent inbox in HostLoom: IInboxStore with one set-if-absent member, InboxStore.FromClaim for a cache-backed store in one line, InMemoryInboxStore, and InboxFilter, appended to the receive pipeline by UseInbox<TStore>(window), UseInbox(factory, window), or UseInMemoryInbox(window). It records {topic}:{subscription}:{messageId} before the handlers run, skips a delivery seen inside the window with an InboxDuplicate payload, runs anyway with an InboxSkipped payload when the store cannot answer, and passes requests through untouched. Metric hostloom.inbox.duplicates; log events 3310 and 3311.
• HostLoomBuilder.ConfigureReceivePipeline(Action<PipeBuilder<ReceiveContext>, IServiceProvider>), for a filter that needs a service from the container, such as a store. The pipeline is still composed once, from the root provider.

Changed

• Source-breaking contract additions from the security hardening below: IsCoordinated on IDistributedLock, ILockHandle, ILeadership, and IScheduleGuard; IOutboxStore.MarkFailedAsync takes the next attempt time and MarkDeadLetteredAsync is new; OutboxMessage.NextAttemptAt, WebSocketTopicDescription.AllowTopicWideSubscription, LeadershipDescription.Coordinated, and SchedulingDescription.GuardCoordinated are new positional record parameters. In-repo implementations and the Testing packages are updated.
• Remote faults no longer carry the handler's exception type and message by default. A handler that wants the caller to see a message throws RemoteFaultException; the previous behavior is available through HostLoomOptions.IncludeFaultDetails. Framework-routed faults use the stable types HandlerNotFound and ResponseTypeMismatch.
• A leader elector whose lock is disabled (Locking:Enabled=false) no longer leads on every instance. LeadershipOptions.WhenUncoordinated defaults to Follow; a single-instance or development host opts into Lead. The elector warns once per role either way.
• A logging scope hole such as {User} destructures a non-scalar value through the same fail-closed, mask-aware path as an event hole instead of calling ToString(); {$User} keeps the plain text form. The templated scope text is rendered from the captured fields.
• The inbox key is length-prefixed per component, so keys recorded before the upgrade are not recognised for the remainder of their window.
• Redis health-check descriptions report reachability and latency without endpoints, machine name, or process id; Describe() is unchanged for logs.
• The development docker-compose.yml binds RabbitMQ, Kafka, and Redis to loopback only.
• RabbitMQ defaults to versioned, role-qualified queue names that separate request routes and event subscription pairs. Existing deployments must follow the queue migration guide or explicitly set RabbitMqOptions.QueueNaming to RabbitMqQueueNaming.Legacy during rollout.
• RabbitMQ event publication waits for publisher confirmation and sets persistent delivery when DurableTopics is enabled. Unconfirmed outbox messages remain retryable; retries can duplicate an event whose confirmation was lost.
• The WebSocket gateway ends a subscription before it answers an invalid credit or ack frame with a fault, so a fault after subscribed is terminal on both peers. Previously the subscription stayed live behind the fault and kept delivering events on a stream the browser client had already discarded, until its orphan cleanup sent an unsubscribe.

Fixed

• Lock acquisition rejects a successful backend reply that arrives after the usable lease has expired, reporting LockFailureKind.Timeout instead of granting an expired handle.
• Concurrent lock renewals, including automatic heartbeats, serialize backend calls with local deadline updates. Waiting honours cancellation and rechecks ownership before issuing a command, preventing an older renewal from overwriting a newer lease deadline.
• Kafka reply consumers retain progress across reassignments and fail initialization instead of falling back to an unresolved end offset. Live request/reply tests provision their owned topics.
• RabbitMQ subscriptions close channels even when cancellation callbacks throw and tolerate repeated disposal.
• Redis and Valkey validate canonical tagged-write domains before backend I/O, preventing values from being accepted and then orphaned by the tag invalidation filter.
• Logging bounds message and text-field buffer growth before encoding and falls back to the capped rendered message when a CLEF template exceeds the message budget.
• The npm publish step uses one environment mapping for its version and bootstrap token.
• Lock-loss callback exceptions no longer escape timer callbacks or interrupt lease accounting.
• Valkey tag invalidation removes only snapshotted members, retaining concurrently added keys for later invalidation. Restricted cache ACLs now also need SREM permission.
• Concurrent L1 mutations keep byte accounting consistent with the stored entries.
• WebSocket subscription initialization and removal detach linked cancellation resources safely.
• Pending leader-channel writes wake on leadership loss and follow the follower discard policy.
• Scheduled jobs retain stop, timeout and claim-loss outcomes after cooperative normal returns.
• Logging shutdown bounds synchronous sink disposal and cancellation callbacks as well as asynchronous stalls.
• A lock heartbeat that fails re-arms itself halfway to the lease end and keeps retrying until an extension succeeds or the lease runs out. Previously one transient backend failure ended automatic extension for good and the lease expired under a healthy action.
• A release that throws is logged as LockReleaseFailed only: the backend keeps the owner's lease until it expires, so it is no longer reported as a loss, counted on hostloom.lock.lost, or allowed to cancel LostToken. LostToken also stays readable after the handle is disposed.
• RedisConnection reports the client name of a multiplexer supplied by Redis:ConnectionFactory, so tracking finds that connection's subscriber. Previously the channel looked for the suffixed name it would have configured itself, never found it, and fell back to the explicit channel. Tracking now refuses a client name shared by more than one pub/sub connection on a server instead of redirecting to whichever the server listed first, and broadcast mode checks notify-keyspace-events where CONFIG GET is permitted instead of reporting itself enabled on the strength of a subscription alone.
• A tagged Redis write pipelines the tag memberships before the value, so a connection lost part-way leaves an index entry for an absent key rather than a value no tag removal reaches.
• CacheFilter runs the rest of the pipe when the lookup finds a remembered null instead of failing on a null payload, and the produced payload replaces the remembered absence.
• The IDistributedCache adapter applies Caching:MaxPayloadBytes to the consumer's bytes; a larger payload is not written, logged at error level once per key per interval, and counted as a payload error.
• The Valkey invalidation channel flushes the in-process tier when its first subscription lands after failed attempts, not only after a lost one, and publishes a probe to itself every InvalidationProbeInterval (30 seconds by default) so a subscriber socket that dies without a close is replaced within one CommandTimeout instead of looking subscribed indefinitely.
• RecordingCacheStore wraps a store whose channel is a separate class, as FaultingCacheStore already did, instead of failing when the cache subscribes.
• The cache tracks invalidation generations in 1024 key stripes, so an invalidation of one key no longer suppresses every in-flight fill on the instance; tag and flush invalidations still suppress all of them, and bulk reads and warmup stay on the whole generation. The echo of this instance's own publish is recognised within Caching:Invalidation:Timeout and counted as echoed on hostloom.cache.invalidations instead of being applied again, so the refill that follows a removal is cached rather than dropped. A distributed read that completes after a later write to the same key no longer replaces that write's in-process entry.

Security

• The WebSocket gateway refuses a keyless subscribe to a keyed topic with forbidden unless the registration passes allowTopicWideSubscription: true. Previously any principal who passed a key-ignoring topic policy received every key's events and the whole snapshot. Keyed examples now use TopicKeyPolicy.SubjectOnly.
• The gateway drops an event that exceeds MaximumMessageSize instead of aborting every subscriber, faults a stream whose snapshot value is too large with snapshot_failed, and answers an oversized response with a message_too_large fault. Only the outbound budget still aborts a connection.
• HostLoomWebSocketOptions.SnapshotInitializationTimeout (30 seconds) bounds how long a subscriber can hold a snapshot provider open by withholding credit; the stream faults with snapshot_stalled and the provider scope is disposed.
• HostLoomWebSocketOptions.MaximumRequestsPerSecond (100) budgets request frames before any scope, authorization, or deserialization work; every other client frame kind shares the control budget. Session expiry works for any lifetime, including DateTimeOffset.MaxValue, and cleanup runs even when the expiry timer faults.
• The gateway resolves every configured authorization policy name at startup and fails the host with the missing names; a runtime authorization exception maps to forbidden.
• Kafka validates the hostloom-reply-to header against topic-name rules and, when KafkaOptions.AllowedReplyTopics is set, against that list before the handler runs. A reply that cannot be produced is logged as UnroutableReplyException, committed, and skipped; the handler is not re-run. The reply consumer starts from the latest offset, and KafkaOptions.MaxRequestAge can reject stale records.
• KafkaOptions gains SecurityProtocol, SaslMechanism, SaslUsername, SaslPassword, SslCaLocation, and ConfigureClient so the transport can authenticate and use TLS; the Kafka guide documents a secure baseline.
• RabbitMQ accepts only server-named reply queues (amq.gen-*, amq.rabbitmq.reply-to) unless RabbitMqOptions.AllowNamedReplyQueues is set, nacks with requeue when a delivery is cancelled by shutdown, and declares queues with x-dead-letter-exchange when RabbitMqOptions.DeadLetterExchange is set.
• Envelope decoding rejects an empty MessageId, an empty CorrelationId, and a response or fault without one, so a sender cannot poison the inbox window with a zero id.
• The outbox retries with exponential backoff (OutboxOptions.RetryDelay, MaxRetryDelay, RetryBackoffFactor) and dead-letters after OutboxOptions.MaxAttempts (10), counted on hostloom.outbox.dead_lettered; the persisted failure text is the exception type name only.
• The messaging.message.type tag is unknown for an unregistered type, bounding cardinality.
• [NotLogged] and [LogMasked] are honoured on overridden virtual properties. HostLoomLoggerOptions.MaxMessageLength (16 KiB) and MaxTextFieldLength (8 KiB) cap message text, plain fields, and scope texts.
• The Redis invalidation channel drops a message over 1 MiB, over 10 000 keys and tags, or with an invalid key, counted on hostloom.redis.invalidation.malformed; the Valkey codec validates each item the same way. Tag members outside the namespace's data prefix are removed from the index instead of unlinked, counted on hostloom.redis.tag.members_rejected and hostloom.valkey.tag.members_rejected. The Redis guide documents the ACL and TLS baseline.
• The single-flight guard map is bounded at four times L1.MaxEntries and falls back to striped semaphores, and the IDistributedCache adapter validates every key.
• CacheKey.FromSensitive(value, key) and LockKey.FromSensitive(value, key) hash with HMAC-SHA256 for low-entropy secrets; the unkeyed form is documented for high-entropy inputs.
• LeaderChannelOptions.DrainOnLoss drops buffered leader-era items when leadership is lost. The elector loop survives an unexpected exception (hostloom.leader.loop.faults, backoff, and continue) and validates its delays against the timer limit. UseGuard, AddHostLoomLeadership, and AddRole throw when an earlier registration would silently supersede the chosen guard or leadership. Cron steps are bounded to the field range.
• Release workflows pass tag and version values to shell steps through environment variables.