HostLoom 0.10.0

The tenth release of HostLoom. Five threads run through it. The first is transport recovery. RabbitMQ publications now borrow confirmed channels from a bounded pool instead of sharing the reply channel behind a single gate, so one stalled confirmation no longer serializes every publication, and RabbitMqOptions.PublishTimeout bounds an event publication end to end, from waiting for a channel through the broker's confirmation. A channel whose publication was not confirmed is never reused: it closes in the background, capped at MaxConcurrentPublishes and reported by hostloom.rabbitmq.channels.closing, so a timed-out publication returns at its deadline rather than after the broker's close reply, which a real-broker test measured at twenty seconds before the change. RequestDispatchConcurrency and EventDispatchConcurrency let a request listener handle up to sixteen deliveries at once while an event subscription keeps its queue order by default, and a rejected delivery is logged with the exception that caused it. Kafka's reply consumer is now its own component that recovers when its first watermark query fails, rather than latching the failure until the host is recreated; commits run from the consumer loop after the handler returns, so a failed commit no longer re-runs a handler whose reply was already produced; and an event handler that keeps failing holds its partition and retries instead of being dropped after five attempts. The in-memory transport now behaves like a broker where tests depend on it: handlers receive the listener's token rather than the caller's, a subscriber failure is logged instead of failing the publisher and making an outbox retry every subscriber, an unbound request waits for its timeout, and disposal ends every pending request.

The second thread is a transport error contract. IRequestBroker.RequestAsync and IEventBroker.PublishAsync now throw MessagingTransportException for every transport failure other than those with their own documented type, carrying the address and the client library's exception as its inner exception, so an application no longer catches Kafka or RabbitMQ types to handle an ordinary broker failure. The broker contracts document what a caller observes: a timeout ends with RequestTimeoutException and an unbound or unroutable address times out, caller cancellation carries the caller's own token, disposal throws ObjectDisposedException, a handler's token belongs to its listener, a completed publication means acceptance rather than handling, and a health probe states whether it reports local state or checks the broker. A shared conformance suite runs the same fifteen scenarios on the in-memory transport, RabbitMQ, and Kafka, and it found two gaps that this release closes: a cancelled RabbitMQ caller received the transport's linked token instead of its own, and disposing the in-memory transport left a pending request waiting.

The third thread is cache invalidation. Redis tracking now flushes the in-process tier after tracking is re-registered on a restored command connection, because CLIENT TRACKING is per connection and an interactive reconnect alone lost invalidations. A tagged conditional write sends its value and tag memberships in one script, a Redis Cluster connection without hash tags on database zero is refused before any cache command, and the Valkey invalidation channel probes its subscriber over a private channel with its own queue, keeps its reconnect backoff while a socket flaps, and flushes on its first acknowledged subscription. Invalidation generations are striped by tag as well as by key, so a tag invalidation discards only the in-flight fills that declare a tag sharing its stripe, and a key with a slow factory is cached again while unrelated tags are being invalidated. That race logic now lives in its own component with synchronous unit tests, which took the tiered cache from 1,740 to 1,434 lines without changing its public surface. The caching reference now states how a write becomes visible on other instances in each invalidation mode, and cache conformance gains cross-instance removal, overwrite, own-echo, delivery-gap, and tag-removal scenarios on the in-process store, Redis, and Valkey.

The fourth thread is coordination under uncertainty. A lock acquisition can end without a certain outcome: the caller cancels, MaxWait expires with the call in flight, the reply arrives after the usable lease, or the provider times out after the server applied the write. The lock kernel now gives each acquisition its own provider token and follows every such ending with exactly one bounded, owner-checked release that the caller never waits for, counted on hostloom.lock.orphan_releases, so an abandoned grant no longer holds the key for a full lease; providers follow standard cancellation. A leader whose renewal fails while its lease is still held now stays leader and retries until an extension succeeds or the lease ends, instead of handing leadership over on one transient provider failure, and a hosted elector whose lease exceeds Locking:MaxLease fails at startup rather than being capped silently.

The fifth thread is evidence. HostLoom.Transport.RabbitMq and HostLoom.Transport.Kafka meters count publications, deliveries, rejections, requeues, commits, rewinds, skips, loop faults, reply-consumer starts, and pending requests, and the Kafka meters are asserted against a real broker. Opt-in experiments time out a RabbitMQ publication and drop and restore its connection through a private TCP proxy, and pause a Kafka container the test owns, including during the reply consumer's watermark query, which gives the Kafka recovery its first real-broker proof.

Upgrading from 0.9.0 changes behaviour an application can observe, each stated under Changed below. Code that caught a Kafka or RabbitMQ client-library exception from RequestAsync or PublishAsync must catch MessagingTransportException and read its inner exception. RabbitMQ request handlers on one endpoint run up to sixteen at a time, so a handler that is not safe to overlap needs RabbitMqOptions.RequestDispatchConcurrency = 1 or a receive-pipeline concurrency limit, and a deployment with a low channel_max should budget the pooled publisher channels. A Kafka event handler that keeps failing now holds its partition until it succeeds. A Redis Cluster deployment must set UseHashTags = true and DatabaseIndex = 0, which renames its keys, so expect a cold cache and roll every instance together where locks must stay exclusive. A Valkey ACL that names the invalidation channel must also allow its :probe:* suffix. Cancelling a lock acquisition no longer stops a command already sent, and a hosted elector whose lease exceeds Locking:MaxLease now fails at startup.

No package is published for the first time; the set is the same as 0.9.0. @hostloom/websocket-client is not part of this NuGet set; that package carries its own version and ships when a websocket-client-vX.Y.Z tag is published.

Added

• RabbitMqOptions.PublishTimeout (30 seconds) bounds an event publication end to end: waiting for a publisher channel, declaring the exchange, and the broker confirmation. When it elapses the publisher receives a TimeoutException; a request publication stays bounded by the request timeout, which now also covers client initialization. RabbitMqOptions.MaxConcurrentPublishes (16) caps the confirmed publisher channels the broker keeps in its pool; each outstanding publication owns one exclusively. Both are validated when the broker is constructed. A channel whose publication was not confirmed is never reused and closes in the background, so the caller returns at its deadline rather than after the broker's close reply. At most MaxConcurrentPublishes channels close at once, and a publication that needs a new channel meanwhile waits within its own deadline. The hostloom.rabbitmq.channels.closing gauge reports them, and broker disposal waits at most five seconds for them before logging RabbitMqChannelsStillClosing (1403) and closing the connection.
• RabbitMqRequestBroker(IOptions<RabbitMqOptions>, ILogger<RabbitMqRequestBroker>?). A delivery whose handler throws and whose requeue also fails is logged as RabbitMqDeliveryRejected (1401) at Error with the delivery tag, the original exception, and whether DeadLetterExchange is configured, before it is rejected without requeue. Previously the failure was swallowed. Logging does not retain the message; configure DeadLetterExchange for that.
• KafkaRequestBroker implements IBrokerHealthProbe, so the readiness check no longer treats Kafka as always reachable. It reports unhealthy after disposal, after a failed reply-consumer initialization ("the next request retries initialization"), and while the reply consumer is still awaiting its partition assignment; otherwise healthy, stating that broker reachability is not probed. It reflects local consumer state only.
• InMemoryRequestBroker(ILogger<InMemoryRequestBroker>?). A subscriber that throws is logged as InMemoryEventFailed (1402) at Error with the subscription name.
• The transports reference documents the two RabbitMQ options, the probe support of each transport, and the in-memory delivery rules below.
• RabbitMqOptions.RequestDispatchConcurrency (16) and RabbitMqOptions.EventDispatchConcurrency (1), the deliveries a request listener and an event subscription handle concurrently on their channel, each validated between 1 and PrefetchCount. Previously both were the client library's default of 1, so PrefetchCount only buffered deliveries and every handler on an endpoint ran in turn.
• InMemoryRequestBroker(ILogger<InMemoryRequestBroker>?, TimeProvider?). The request timeout and the unbound-address wait run on the supplied clock, and a host that registers a TimeProvider before UseInMemory() drives them from a test clock without waiting in real time.
• Meters HostLoom.Transport.RabbitMq and HostLoom.Transport.Kafka. RabbitMQ counts hostloom.rabbitmq.publishes by outcome (confirmed, returned, failed, timed_out) with hostloom.rabbitmq.publish.duration, hostloom.rabbitmq.deliveries.rejected by reason (malformed, handler_failed), hostloom.rabbitmq.deliveries.requeued, hostloom.rabbitmq.connections by event (opened, recovered), and the hostloom.rabbitmq.requests.pending gauge. Kafka counts hostloom.kafka.produced by kind (request, reply, event), hostloom.kafka.consumed, hostloom.kafka.committed, hostloom.kafka.records.skipped by reason (malformed, unroutable_reply, attempts_exhausted), hostloom.kafka.records.rewound, hostloom.kafka.loop.faults by stage, hostloom.kafka.reply_consumer.initializations by outcome, and the hostloom.kafka.requests.pending gauge. Handler outcomes stay on the kernel's request instruments and are not counted twice.
• hostloom.lock.orphan_releases, tagged hostloom.lock.outcome (released, absent, failed), and the LockOrphanRelease (3108) Debug event, which record the release described under Changed below.
• MessagingTransportException, thrown by IRequestBroker.RequestAsync and IEventBroker.PublishAsync when a transport cannot carry a request or an event. It carries the Address and the client library's exception as InnerException. Whether the broker accepted the message is unknown when it is thrown, so a retry can deliver an event twice or run a handler twice.
• The broker contracts document their observable outcomes: a timeout ends with RequestTimeoutException and an unbound or unroutable address times out, caller cancellation carries the caller's own token, disposal throws ObjectDisposedException, a handler's token belongs to its listener, a completed publication means acceptance rather than handling, and a health probe states whether it reports local state or checks the broker. A shared transport conformance suite runs the same scenarios on the in-memory transport, RabbitMQ, and Kafka.
• FaultingLockProvider.HoldReplies(), DeliverReplies(), and WaitForHeldReplyAsync() hold acquisition replies so a test can abandon an attempt the backend already granted; Heal() also delivers held replies. Lock conformance gains AbandonedAcquisition_LeavesTheKeyFree, run on the in-process lock, Redis, and Valkey.
• The caching reference documents write visibility across instances for each invalidation mode: SetAsync and SetIfAbsentAsync publish nothing, so with the explicit channel another instance keeps its old in-process value until it expires, while tracking and broadcast observe store writes. In broadcast mode the writer also evicts its own entry on each write. Cache conformance gains cross-instance removal, overwrite, own-echo, delivery-gap, and tag-removal scenarios.
• Opt-in real-broker experiments: HOSTLOOM_RABBITMQ_CHAOS=1 times out a publication and drops and restores a connection through a private TCP proxy, and HOSTLOOM_KAFKA_CHAOS=1 pauses a Kafka container the test owns, including during the reply consumer's watermark query. The Kafka meters are also asserted against the compose broker by default.

Changed

• RabbitMQ requests and events are published on a pool of confirmed channels, up to MaxConcurrentPublishes, and the exclusive reply queue keeps its own channel. Previously every request and event shared the reply-queue channel behind a single gate, so one stalled confirmation serialized all publication. An upgrader who needs the old serialization sets MaxConcurrentPublishes = 1; a deployment with a low channel_max should budget the extra channels per instance. Publishing an event no longer initializes the reply queue. Disposal cancels publishers that are waiting for a channel or a confirmation with ObjectDisposedException, waits for borrowed channels to return and at most five seconds for channels still closing, then closes the pool, the client channel, and the connection, aggregating any failures; repeated disposal shares one task.
• Kafka commits an offset after the handler returns, from the consumer loop, rather than inside the handler. Successful handling and the commit are separate failure boundaries: a commit that throws is logged and does not seek back and re-run a handler whose reply was already produced. Kafka can still redeliver an uncommitted record after a reassignment, so handlers must remain idempotent.
• A Kafka event subscription retries a record whose handler keeps throwing indefinitely, seeking back with the consume backoff each time, instead of logging an error and committing past it after five attempts. A record that never succeeds holds its partition at that offset until shutdown, so a handler that wants a poison event skipped must swallow it itself. Request subscriptions keep the five-attempt cap; malformed records and unroutable replies are still logged, committed, and skipped.
• The in-memory transport delivers like a broker. A handler runs on the thread pool with the listener's or subscription's lifetime token rather than the caller's: caller cancellation ends the caller's wait without cancelling accepted receiver work, and disposing the listener, as a host stop does, cancels work in progress instead of letting it finish. The requester of that work waits out its own timeout and gets RequestTimeoutException, as with a broker that has no other listener, instead of seeing the listener's cancellation. A publication is accepted once every subscriber has been attempted; a subscriber failure is logged at 1402 and no longer reaches the publisher as an AggregateException, so an outbox relay does not retry it. A request to an address with no listener waits the full request timeout before RequestTimeoutException, with an inner TimeoutException; previously it was refused at once. Disposing the transport ends every waiting request at once with ObjectDisposedException, every member throws it afterwards, and the probe reports unhealthy. Tests that relied on the old behaviour changed with it: MessagingChaosTests now expects a request whose listener stops to time out and an unbound request to take its budget, and PublishSubscribeTests no longer expects a failing subscriber to fault the publish.
• RedisConnection throws ArgumentException ("Redis Cluster requires UseHashTags = true and DatabaseIndex = 0") whenever it hands out a multiplexer, including one supplied through Redis:ConnectionFactory, if any endpoint reports Cluster topology and the options differ. FailFast = false no longer downgrades this to RedisUnreachableAtStartup (1304): hosted startup fails. An upgrader on Cluster must set UseHashTags = true and DatabaseIndex = 0. Hash tags wrap the namespace segment in {…}, so keys written before the change are under different names: expect a cold cache, and roll every instance together where locks or leases must stay mutually exclusive across the fleet.
• The Valkey invalidation channel publishes its liveness probe to a private channel, {invalidation channel}:probe:{id}, subscribed on the same socket with its own reader and queue, so a slow invalidation handler or an overflowed invalidation queue no longer reads as a dead subscriber. An ACL that names the invalidation channel must also permit the :probe:* suffix. Probe payloads that an earlier version still publishes on the invalidation channel are ignored.
• The Valkey reconnect backoff is reset only after a subscription has stayed established for 30 seconds. Previously any acknowledgement reset it to 100 ms, so a socket that was acknowledged and dropped repeatedly reconnected at the floor rate for as long as it flapped.
• Every acknowledged Valkey subscription, including the first one of a channel's lifetime, hands CacheInvalidation.Flush to its subscribers when FlushLocalOnReconnect is set. Previously the first subscription flushed only if an earlier attempt had failed, leaving entries filled while the acknowledgement was pending. An internal TimeProvider constructor drives the backoff in tests.
• The lock gives each acquisition its own provider token, cancelled only when the lease has run out since the request was sent or when the DistributedLock is disposed; the caller's token and MaxWait end only the lock's wait. An acquisition whose outcome is uncertain no longer leaves a grant held for the whole lease: exactly one bounded, owner-checked release that the caller never waits for follows caller cancellation or MaxWait expiry once the grant arrives, a reply rejected because the usable lease had already run out, and a provider failure of kind Timeout or Other, where the caller still receives the same exception. Unavailable means the backend was not reached and releases nothing, which includes a Redis connection that drops mid-command. Cancelling therefore no longer stops a command already sent: an abandoned attempt can cost one extra write plus one release, and a provider call can outlive its caller by up to one lease. Providers follow standard cancellation, and the locking reference no longer claims that cancellation leaves nothing held.
• Disposing a DistributedLock ends its in-flight acquisitions with ObjectDisposedException, and every later TryAcquireAsync or ExecuteWithLockAsync throws it, even with locking disabled. Held handles still release.
• FaultingLockProvider.TryAcquireAsync is asynchronous, so an injected failure arrives as a faulted task instead of a synchronous throw.
• A leader whose renewal fails while its lease is still held stays leader and retries at the renewal interval or half the remaining lease, whichever is sooner, never more often than a twentieth of the lease, until an extension succeeds under the same term or the lease ends. Previously one transient provider failure handed leadership over and cancelled leader-only work. A refusal, a lost handle, lease end, resignation, and stop still end leadership at once. Such a failure is recorded as failed rather than refused on hostloom.leader.renew.duration.
• A hosted elector whose Leadership:Lease exceeds Locking:MaxLease fails at startup with a message naming the role and both values. Previously the lock capped the lease silently, which could break the rule that the renewal interval is at most half the lease. An elector built without the container is still capped silently.
• RequestAsync and PublishAsync report transport failures as MessagingTransportException. RabbitMQ maps nacks, closed connections and channels, an unreachable broker, and other client-library, socket, and I/O failures; Kafka maps a reply consumer that fails to start and ProduceException. An unroutable request still ends in RequestTimeoutException, PublishTimeout still throws TimeoutException, and a failure after disposal is ObjectDisposedException. ListenAsync and SubscribeAsync still surface the client library's exception, which fails host startup. A cancelled RabbitMQ caller now receives its own token instead of the transport's linked one, and a Kafka publication after disposal throws ObjectDisposedException.
• Invalidation generations are striped by tag as well as by key. A tag invalidation suppresses only in-flight fills that declare a tag sharing its stripe, an untagged fill ignores tag invalidations, and a fill on an unrelated tag reaches both tiers. Previously any tag invalidation on any instance discarded every in-flight fill, so a key with a slow factory was never cached while tags were being invalidated. A flush still suppresses every fill, bulk reads and warmup stay on the whole generation, and a tagged payload read from the distributed tier while a tag invalidation was applied is returned but not copied into the in-process tier. Every writer of a key must declare the same tags for the suppression to hold.
• StackExchange.Redis 3.3.1 and FusionCache 2.9.0 (with its System.Text.Json serializer).

Fixed

• A Kafka reply consumer whose initial watermark query failed no longer poisons every later request. The failed consumer is disposed, its offsets cleared, and the next request starts a fresh consumer; previously the failure was latched and the documented remedy was to recreate the host. Disposal waits for an initialization in flight before closing.
• A tagged Redis SetIfAbsentAsync writes the tag memberships and the value in one script, and a losing writer adds no membership. Previously the conditional SET and the membership batch were two round trips, so a disconnect or caller cancellation between them left a value no tag removal reaches. The script also refuses a tag index that is not a set before it writes.
• Redis tracking flushes the in-process tier after the command connection is restored and tracking is registered on the new connection, when FlushLocalOnReconnect is set and the mode is not Broadcast; a fallback to the explicit channel keeps the flush pending until tracking registers. Previously only a re-established subscription connection flushed, on the 0.7.0 reasoning that an interactive blip loses nothing, which does not hold for CLIENT TRACKING: it is per connection, so keys read before the drop had no invalidation.
• Disposing a stale in-memory listener or subscription handle removes only its own registration: a successor bound to the same address or subscription name after the first disposal is no longer removed by disposing the first handle again.
• The in-process tier's warning when it clears itself at 150 % of L1.MaxEntries (CacheL1Cleared, 1101) is logged through the cache's logger; previously the tier was built without one and the warning never appeared. Expired entries are swept once per interval instead of twice.
• The Valkey invalidation channel throttles its warnings on the injected clock and is safe when several threads warn at once.