HostLoom 0.11.0

The eleventh release of HostLoom. Five threads run through it. The first is delivery that could quietly go wrong. The idempotent inbox recorded a delivery's key before its handlers ran and never released it, so a handler failure, a cancellation during a rolling deploy, or an inbox placed inside UseRetry turned the broker's redelivery into an acknowledged duplicate and lost the event; a failed or cancelled run now releases its key through the new IInboxStore.ReleaseAsync, and on a real Kafka broker 500 events with 148 failing first attempts were all handled exactly once, where 0.10.0 lost exactly those 148. An envelope whose messageType was JSON null passed validation and failed the handler lookup as an argument error, which a Kafka event subscription retried forever and so stalled its partition; missing message and response types and incomplete faults are now rejected as malformed and skipped as poison. Kafka replicas configured with one ClientId shared a reply group and split the response partitions, so most of their requests timed out; the reply group is now named per instance. The outbox relay used to charge every pending message an attempt on every drain during a broker outage, so a 70-second RabbitMQ outage dead-lettered 607 of 1,000 events; it now stops a drain at the first failed publish, backs off as a whole, probes with one message, and republishes what it holds as soon as a publish succeeds, which took the same outage to no dead letters and 12 failed attempts. A failed mark after a successful publish is no longer counted as a failed attempt, a second inbox or outbox registration throws instead of silently doubling or dropping, a request that a receive filter completes without its handler is answered with a HandlerNotRun fault, failed event handlers are counted as faults, and host shutdown waits no longer than the host allows.

The second thread is the RabbitMQ and Kafka transports. Stopping a RabbitMQ listener or subscription waited for the broker to confirm each channel close, which against a stalled broker could hold host shutdown for twenty seconds per consumer; a stop now stops the broker's deliveries, waits up to five seconds for the handlers it cancelled with the channel still open, so a handler that finishes anyway is acknowledged rather than reported as a rejection, and closes the channel in the background. Disposing the transport against an unresponsive broker returns after about seven seconds instead of about thirty. A consumer the broker cancels, for example because its queue was deleted, used to stop consuming silently; it is now logged, counted on hostloom.rabbitmq.consumers, and restored by declaring its queue again on a new channel. Foreign clients that use RabbitMQ direct reply-to are answered, which the documentation promised but the exact-name check prevented; a Kafka publication still waiting for its delivery report when the transport is disposed ends with ObjectDisposedException instead of hanging, as it did on a paused broker in 0.10.0; and every line either transport logs now carries a stable event id.

The third thread is the WebSocket gateway. A close the server started, for session expiry, an administrative disconnect, rate limiting, a protocol violation, or shutdown, cancelled the pending receive, which aborts the runtime's socket, so clients saw an abnormal closure instead of the status and reason, and the TypeScript client reconnected after a logout. The writer now sends the frames already queued and the close frame and waits for the client's answer, bounded by the new CloseTimeout, and sessions close with 1001 when the host starts stopping rather than after Kestrel has drained its connections, which took a host stop with connected clients from thirty seconds to milliseconds. The gateway endpoint also accepts the HTTP/2 extended CONNECT handshake, which browsers use on an existing HTTP/2 connection and which previously got 405, and a Protocol Buffers frame the reader cannot follow closes with 1007 while any other unexpected failure closes with 1011 instead of a normal closure.

The fourth thread is correctness elsewhere in the framework. A cron schedule in a zone with daylight saving time ran back to back during the fall-back hour, because an ambiguous local time resolved to an instant before the scheduler's input; each local time now occurs once. A registered pipeline's WithRetry re-ran in the failed attempt's DI scope with the same filter instances, so scoped state such as a unit of work leaked into the retry; every attempt now gets its own scope. A composition rule's Skip, Throw, and Replace collided with the rule's own entries, so ExpectMany().Throw() over two implementations always threw; policies now compare against registrations that existed before the rule. A Redis invalidation channel whose subscription failed left an unread queue attached for every attempt, each buffering every invalidation for the life of the process after recovery; a failed attempt now detaches exactly what it attached.

The fifth thread is logging and release security. AddHostLoomLogging registered its provider as an instance the container never disposed, so records still queued at shutdown were lost and a buffered sink was never flushed; the container now owns and drains it. An exception whose Message getter throws, and any other formatter failure, used to stop the log writer for the rest of the process; a formatter failure now drops only that record. [LogMasked] no longer reveals a value too short to keep at least half of it hidden, and dictionary keys and the per-record byte budget are capped like string values. The NuGet release workflow now grants its publishing identity only to a job that runs no restore, build, or test code, and the browser-client workflow reads no npm token.

Upgrading from 0.10.0 changes behaviour an application can observe; each change is stated under Changed. The inbox releases a delivery's key when its handlers fail or are cancelled, so a custom IInboxStore should implement the new ReleaseAsync, and registering the inbox or the outbox a second time throws. An envelope without a message type, a request without a response type, and an incomplete fault are rejected as malformed, and a request that a receive filter completes without its handler is answered with the new HandlerNotRun fault. A registered pipeline's retry runs every attempt in a new DI scope. A composition rule's policy no longer collides with the rule's own entries. The Kafka reply group is named per instance, so replicas may share a ClientId. A WebSocket close the server starts now completes the close handshake, bounded by CloseTimeout; a RabbitMQ consumer the broker cancels is restored by declaring its queue again; and the bare amq.rabbitmq.reply-to reply address is rejected. Stopping a RabbitMQ listener or subscription waits up to five seconds for its handlers, and a delivery its closing channel cannot settle is logged as a warning and redelivered rather than logged as a rejection. The outbox relay stops a drain at its first failed publish and backs off as a whole, so an outage costs one attempt per failed drain instead of one per pending message. [LogMasked] writes only the mask for a value too short to keep at least half of it hidden, which includes a sixteen-digit number shown as its first six and last four digits, and a formatter failure drops only that record. No package is published for the first time.

No package is published for the first time; the set is the same as 0.10.0. @hostloom/websocket-client is not part of this NuGet set; that package carries its own version and ships when a websocket-client-vX.Y.Z tag is published. Its unreleased changes pace credit and acknowledgements below the gateway's control-frame budget and cancel a connect queued behind a second close; the published 0.2.0 client is closed with rate_limited at ordinary event rates, so upgrade it once 0.3.0 ships.

Added

• HostLoomWebSocketOptions.CloseTimeout (5 seconds) bounds how long a session this side is closing waits for the client's close frame before aborting; the abort is logged as WebSocketCloseTimedOut (4111).
• IInboxStore.ReleaseAsync(string, CancellationToken), a default member that does nothing, and InboxStore.FromClaim(tryRecord, release). InMemoryInboxStore implements the release. A release the store cannot complete is logged as InboxReleaseFailed (3312).
• InMemoryOutboxStore.PublishedCapacity (1,000 by default; zero keeps none) bounds the published messages the in-memory store keeps; it used to keep every frame for the life of the process.
• The hostloom.message.kind tag (request or event) on hostloom.request.duration, .active, .faults, and .retries, and the HandlerNotRun fault type.
• Log events OutboxMarkPublishedFailed (3305), InMemoryHandlersAbandoned (1404), and WebSocketSessionFailed (4112); the WebSocket close reason internal_error; and the logging drop reason format_failed.
• The hostloom.rabbitmq.consumers counter and log events RabbitMqConsumerCancelled (1411), RabbitMqConsumerRestored (1412), and RabbitMqConsumerRestoreFailed (1413).
• Log events RabbitMqDeliveryUnsettled (1405), RabbitMqHandlersAbandoned (1406), RabbitMqConnectionStillClosing (1407), and RabbitMqChannelCloseFailed (1408); stable event ids 1421 to 1430 for every Kafka log line, which used to log event id 0; and the outbox relay's OutboxRelayBackingOff (3306) and OutboxRelayRecovered (3307), each logged once when a relay backoff begins and ends.

Changed

• The inbox releases a delivery's key when its handlers throw or are cancelled, so the transport's redelivery runs them again: a Kafka event that failed, a RabbitMQ delivery requeued by a shutdown, and an inbox inside UseRetry no longer have their next attempt acknowledged as a duplicate. The release runs on its own five-second token and never replaces the handler's exception. A custom store must implement ReleaseAsync for this; one that does not keeps the earlier behaviour. A process that stops between recording a key and completing the handlers still loses that event's redeliveries inside the window.
• An envelope with a null or empty messageType, a request with a null or empty responseType, and a fault whose fault object lacks an error type or a message are rejected as MalformedEnvelopeException. A null messageType on an event used to fail the handler lookup as an argument error, which Kafka retried forever and so stalled the partition; the record is now committed past as poison, and RabbitMQ counts it as malformed rather than handler_failed.
• A registered pipeline's WithRetry gives every attempt a fresh DI scope and fresh filter instances, and evaluates EnabledWhen again. Previously a retry reused the failed attempt's scope, so scoped services such as a unit of work kept that attempt's state. Filter construction and scope disposal now run inside the retry and timeout. The context and its payloads are still shared by the attempts of one run.
• A composition rule's Skip, Throw, and Replace compare its entries only with registrations that existed before the rule: the collection and earlier rules. Entries of the same rule no longer collide with each other, so ExpectMany().Throw() over two implementations registers both instead of throwing, Replace and Skip keep every implementation instead of the last or first, and As(IA, IB).ExpectOne().Replace(ImplementationType) keeps both services. A hand-written plan that gives consecutive entries one origin is treated as one rule.
• The Kafka reply consumer group is named by a random per-instance id instead of KafkaOptions.ClientId, and no longer auto-commits. Replicas configured with one client id used to share a reply group, so the response partitions were split between them and most of their requests timed out. Each process start also stops leaving a committed group behind.
• A second UseInbox or UseInMemoryInbox, or a second UseOutbox or UseInMemoryOutbox, on one service collection throws InvalidOperationException. A second inbox used to wrap the first over the same store, so every delivery looked like a duplicate and no handler ran; a second outbox was silently ignored.
• Stopping the HostLoom endpoints waits no longer than the host's shutdown token; a stop the host gave up on finishes in the background and logs any failure. Stopping an in-memory listener or subscription, or disposing the transport, cancels its running handlers and waits up to five seconds for them, then logs and abandons a handler that ignores cancellation.
• A log formatter exception drops only the record being formatted, counted with reason format_failed, and the writer keeps going; only sink failures fault the pipeline and set WriterFault.
• [LogMasked] and Mask<T> reveal ShowFirst and ShowLast characters only when at least as many stay hidden, and otherwise write the mask text alone. A PIN under ShowLast = 4 used to be written in full; a 16-digit number under ShowFirst = 6, ShowLast = 4 is now fully masked too.
• A RabbitMQ listener answers clients that use direct reply-to, whose ReplyTo the broker rewrites to amq.rabbitmq.reply-to.<token>; they used to be rejected as malformed although the documentation said they were answered. The bare pseudo-queue name, which a listener never receives from a real client, is now rejected.
• Stopping a RabbitMQ listener or subscription stops the broker's deliveries, cancels its handlers, and waits up to five seconds for them before closing the channel in the background; it no longer waits for the broker to confirm the close. A handler that finishes within that wait is answered and acknowledged; one still running is logged (1406) and left running, and the broker redelivers its message. A reply, acknowledgement, or rejection that a closing channel refuses is logged as RabbitMqDeliveryUnsettled (1405, Warning) and counted as requeued, instead of as a rejected handler failure (1401, Error), and a refused rejection no longer escapes into the client library. Disposing the transport waits at most five seconds for closing channels and two for the connection (1407), so against an unresponsive broker it returns after about seven seconds instead of about thirty; the client library finishes the close in the background. Against a stalled broker each channel close could hold host shutdown for the client's twenty-second continuation timeout, one channel after another.
• A RabbitMQ consumer the broker cancels, for example because its queue was deleted, is detected and logged, and its listener or subscription declares its queue again and resubscribes on a new channel, retrying with a doubling wait up to thirty seconds. It used to stop consuming silently. A cancelled reply consumer is replaced by the next request. A queue deleted on purpose while the service runs is therefore re-created; stop the service before removing its queues. Events published after the deletion and before the queue is bound again are confirmed by the broker and dropped by the exchange, as for any topic with no bound queue.
• The outbox relay stops a drain at the first failed publish and charges only that message an attempt. It holds the rest of the claimed batch, untried and uncharged, and publishes it oldest first as soon as a later publish succeeds, provided less than half of Outbox:ClaimLease has passed; after that, or when the relay stops, lease expiry returns those messages to the store. After a failed drain the relay pauses, ignoring wakes and the poll interval, for Outbox:RetryDelay grown by Outbox:RetryBackoffFactor per consecutive failed drain up to Outbox:MaxRetryDelay, then probes with one message, skipping the one that just failed when another is due, and resumes full batches after the first successful publish. A transport outage used to charge every pending message on every drain, which with the defaults dead-lettered the whole backlog after about twenty minutes. Against RabbitMQ with 1,000 pending events, a 70-second outage went from 607 dead-lettered and 6,884 failed attempts to none and 12, and a broker restart from 548 failed attempts to 3, with the backlog cleared 3.4 seconds after the broker served again.

Fixed

• A cron schedule in a zone with daylight saving time no longer runs repeatedly during the fall-back hour. CronExpression.GetNextOccurrence returned the earlier instant of an ambiguous local time even when it preceded its input, so the scheduler found every occurrence overdue; an ambiguous local time now occurs once, as documented.
• AddHostLoomLogging registers its provider so the container owns it: disposing the host drains the queue and flushes and disposes the sink. Previously the provider was never disposed, so records still queued at shutdown were lost and a buffered sink was never flushed. The writer thread now starts when logging is first resolved rather than at registration.
• An exception whose Message getter throws no longer faults the log writer and silences every later record. JsonLogFormatter writes [MessageUnavailable] for it, and caps error.message at its maxExceptionLength like the stack trace.
• The WebSocket gateway completes the close handshake for every close it starts. Session expiry, IWebSocketSessionControl disconnects, rate limiting, protocol violations, and host shutdown used to cancel the pending receive, which aborts the runtime's socket, so clients saw an abnormal closure (1006) instead of the status and reason; the TypeScript client then reconnected after a logout or revocation. The writer now sends the frames already queued and the close frame, and the session ends when the client answers or CloseTimeout elapses. A disconnect or shutdown waits up to CloseTimeout for each client that does not answer.
• Host shutdown closes WebSocket sessions with 1001 when the host starts stopping, before Kestrel drains its connections, instead of after it; a WebApplication with connected clients no longer waits out HostOptions.ShutdownTimeout on every stop. A session accepted after shutdown began is closed without a welcome.
• A Redis invalidation channel whose subscription fails no longer leaves a StackExchange.Redis queue attached for every failed attempt. Each attempt's queue stayed subscribed without a reader, so after Redis recovered every invalidation was buffered in each of them for the life of the process, and disposal did not detach them. Each subscription now attaches one handler, a failed attempt detaches its own, disposal detaches exactly what the channel attached and leaves other subscriptions on a shared multiplexer in place, and a broadcast retry subscribes only the keyspace patterns still missing. Messages are applied by one reader, and a message that fails to apply is logged as RedisInvalidationMessageFailed (1320) instead of ending delivery.
• A failed MarkPublishedAsync after a successful publish no longer counts an attempt, backs off, republishes, and finally dead-letters a delivered message. It is logged (3305) and the claim lease expires, so the next claim publishes the message again.
• A receive filter that completes a request without running its handler answers with a HandlerNotRun fault instead of an empty success that the client reported as MalformedEnvelopeException.
• A failed event handler is counted in hostloom.request.faults.
• Destructured dictionary keys are capped at MaxStringLength, and MaxEncodedBytesPerRecord is enforced after every element: the element that would exceed it is replaced by the truncation marker. A 1 MB key used to produce a field over 1 MB.
• The WebSocket gateway endpoint accepts the HTTP/2 extended CONNECT handshake as well as the HTTP/1.1 GET upgrade; a browser reusing an HTTP/2 connection used to get 405.
• A Protocol Buffers frame with a negative or implausible length, or with groups nested past protobuf-net's limit, closes the WebSocket session with 1007, and any other unexpected failure while decoding or handling a frame closes it with 1011 internal_error. Both used to end as a normal closure, with an error logged on every attempt.
• Disposing the Kafka transport ends a publication still waiting for its delivery report with ObjectDisposedException instead of possibly waiting forever. As with any transport failure, the event may still have been delivered.

Security

• The NuGet release workflow grants id-token: write and attestations: write only to a publish job that downloads the tested packages and runs no restore, build, or test code; the build job can only read the repository. The publish job checks the package list against the packed packages in both directions before the first push and pushes with --skip-duplicate, so a rerun can finish a partial release.
• The WebSocket client release workflow no longer reads an npm token and refuses to publish from a branch whose name looks like a release tag.