GH-3590's reasoning applies to competing-consumer listeners too: inbox recovery is silently skipped when the database's durability agent lands on a node that does not listen to the envelope's queue

[leonardomaza-prisma]

Versions

• Observed in production on WolverineFx 6.32.0 (PostgreSQL transport + PersistMessagesWithPostgresql(...).RegisterTenants(...), database-per-tenant).
• The code path is unchanged in 6.38.0 — I diffed
  CheckRecoverableIncomingMessagesOperation, EndpointCollection.FindListenerCircuit and
  RecoverIncomingMessagesCommand.DeterminePageSize between V6.32.0 and V6.38.0 and they are
  identical for the branches below.

Setup

A Kubernetes deployment with queue-segregated worker pools. Each pool listens to a different
subset of the PostgreSQL queues, and two more nodes publish but never listen:

Node	Listens to
pool-a	queue-1, queue-2, queue-3
pool-b	queue-4
pool-c	(no PostgreSQL queues — local queues only)
publisher ×2	(none — publisher only)

All queues are ListenerScope.CompetingConsumers. Message storage is multi-tenanted by database
(five tenant databases plus the main store).

Everything below is about queue-1, which only pool-a listens to — the shape that matters is
that a queue has a listener on some nodes and not others.

What happens

Every node publishes the same single capability, wolverine://durability-agents-enabled/, so the
leader considers all five nodes equally eligible and distributes the per-database durability agents
by load. Nothing in that decision knows which queues a node actually listens to.

When a database's durability agent lands on a node that has no listener for a queue, every orphaned
Incoming envelope addressed to that queue becomes permanently unrecoverable, and nothing says so:

// CheckRecoverableIncomingMessagesOperation.PostProcessingCommands()
var listener = _endpoints.FindListenerCircuit(incoming.Destination);
if (listener == null) continue; // This *might* happen during shutdown

The comment assumes the null is transient. In a queue-segregated deployment it is the steady
state — and it is the only branch of that loop that produces no log at all. The three neighbouring
outcomes all log:

Outcome	Log
recovered	"Issuing a command to recover {Count} incoming messages…" (Information)
listener latched	"…but the listener is latched" (Information)
DeterminePageSize returns 0	"Unable to recover inbox messages to destination…" (Information)
no listener circuit	— nothing —

Meanwhile the node that does listen to that queue sits idle next to the backlog, and every
queue-depth dashboard reads zero, because the envelopes already left the transport table.

Evidence

Measured in production on 2026-09-15, five tenant databases, same instant, same message types
(all addressed to postgresql://queue-1/), with agent placement as the only variable:

Database	Durability agent on	Listens to queue-1?	Orphans
tenant-a	pool-a	yes	40 → 0
tenant-b	pool-a	yes	6 → 0
tenant-c	pool-a	yes	6 → 0
tenant-d	pool-a	yes	6 → 0
(before the move) tenant-a	a publisher node	no	40, static for ~4 days

Confirmed in both directions: the same database's backlog is frozen while its agent is on a
non-listening node and drains within seconds of the agent moving to the listening one. Raising
Wolverine to Information on the listening pod shows "Issuing a command to recover…"
appearing exactly when a drain starts, and no log of any kind for the frozen case — which is
what identifies the listener == null branch.

Why the existing escape hatches do not apply

• Durability.DurabilityAgentEnabled is the only capability the leader consults
  (MessageStoreCollection.NodeCanRunDurabilityAgents → node.Capabilities.Contains(DurabilityCapabilityUri)).
  It is binary: a node either runs database agents or it does not. Turning it off on the
  non-listening pods narrows the lottery but cannot express which queues a node can serve — and in
  a deployment where every pool listens to a different subset, no single node is capable of all of them.

• Pinning an agent (AgentRestrictions.PinAgent) does not work for this family. The pin is
  applied through AssignmentGrid.Node.TryAssign, which requires the node to declare the agent's
  own URI among its capabilities:

  public bool TryAssign(Uri agentUri)
  {
      if (_capabilities.Contains(agentUri)) { Assign(agentUri); return true; }
      return false;
  }

  The durability family only ever publishes the generic wolverine://durability-agents-enabled/, so
  TryAssign always returns false and the pin is discarded — silently, leaving
  agent.IsPinned false. Verified by writing a pin row and observing no reassignment, with the
  agent's own URI absent from wolverine_nodes.capabilities.

You already wrote this argument — for the other half

ListeningAgent.startInboxRecoveryIfNecessary():

/// GH-3590: a durable listener that is only ever active on a single node (Exclusive or PinnedToLeader)
/// can not rely on the per-database durability agent to recover its dormant inbox messages, because
/// that agent is assigned per database and routinely lands on a different node. Such a listener owns
/// its own inbox recovery for as long as it is the active listener.
private void startInboxRecoveryIfNecessary()
{
    if (Endpoint.Mode != EndpointMode.Durable) return;
    if (!Endpoint.IsSingleNodeListener) return;          // <-- the gate
    ...
}

That reasoning — "the agent is assigned per database and routinely lands on a different node" —
is not specific to single-node listeners. It holds for any listener that is not present on every
node.

The implicit assumption for CompetingConsumers seems to be that every node listens to every queue,
so whichever node holds the database agent can serve it. That is true of a homogeneous cluster. It is
false as soon as worker pools are segregated by queue — and then a competing-consumer listener is
exactly as absent from the agent's node as an exclusive one, with the same dormant-inbox outcome.

The difference is that the exclusive case is handled, and this one fails silently.

We are aware the gate is probably not free to relax: with N nodes hosting the same durable listener,
N recovery loops would sweep every database. That trade-off is yours to make — we are reporting the
hole, not prescribing the shape.

What we would hope for

In rough order of how much we would value it:

1. Log the skip. Even at Information, the listener == null branch naming which destination it
   walked past would have turned four days of silence into one grep. Cheapest change, and on its own
   it would have been enough for us to self-diagnose on day one.
2. Widen the IsSingleNodeListener gate to «this node hosts a durable listener for this endpoint»,
   so a competing-consumer listener owns its inbox recovery on the nodes where it actually runs — the
   same remedy Durable exclusive listeners: inbox messages with owner_id = 0 are never recovered when the DurabilityAgent runs on a different node #3590 already applies, for the same stated reason.
3. Or let agent placement know about listeners, if you would rather keep recovery with the database
   agent: a per-queue capability, or EvaluateAssignmentsAsync preferring nodes that can serve the
   database's pending destinations.
4. Make pinning work for this family, as the operator escape hatch it is clearly meant to be.

Repro

A failing test against your own suite, in PostgresqlTests/MultiTenancy next to
exclusive_listener_recovery_across_tenant_databases:
competing_consumer_inbox_recovery_across_nodes.

Two [Fact]s that differ in exactly one thing — whether the node that owns the tenant database's
durability agent is also the node listening to the queue. The topology is expressed entirely through
public configuration (Durability.DurabilityAgentEnabled per host), no test-only hooks:

recovered_when_the_agent_lands_on_the_listening_node      PASSED
dormant_when_the_agent_lands_on_a_node_that_does_not_listen   FAILED

Same envelope, same bare queue address, same tenant database, same live and accepting listener. The
test asserts the listening agent really is registered before seeding, so a red result cannot be a
mis-built host.

Happy to open it as a PR — with or without the one-line logging change — whichever you prefer.

[jeremydmiller]

@leonardomaza-prisma Not yet, and I hate the AI generated issues sometimes.

Why are you hitting this? I'm not buying that this is the solution. Why do some nodes not listen to the queue in question? Are you trying to set up some nodes with a different configuration, but trying to convince Wolverine they're the same logical service?

[leonardomaza-prisma]

@jeremydmiller thanks for the quick reply, and I'm sorry about the issue. You're right. I did use AI for the analysis — I tried to do it as carefully as I could, reading your code and our own topology before writing anything, but what came out reads like AI slop and that's on me.

Why we're hitting it: we run several pods against one message store. The API doesn't listen to the background queues, and our worker pods are split into pools that listen to different queues. What we see is that messages for a queue only come back when the durability agent for that database happens to sit on a pod that listens to it. Otherwise they stay there.

Your description is accurate, it just wasn't deliberate. We never set ServiceName, and all of them share one message store (Postgres, database per tenant). Is there a way to make them separate logical services while sharing that store? That's the part I couldn't work out.

How would you configure Wolverine for this?

[jeremydmiller]

My consistent advice is to make them different logical apps with a different ServiceName and don't share the message store schema.

[jeremydmiller]

I'm moving this to a discussion

[arical]

We're hitting a very similar issue. We run wolverine on our API solely for publishing events to queues. We have competing consumers that pick these events up and process them. The API has no listeners or subscribers and we don't want it to pick up any events.

Is there a way to set an app as a publisher, not a consumer, that I'm missing? The API is currently picking up events and not being able to process them, which is problematic.