Stuck consumer causes silent message loss on in-memory bus

[MatthewKyle22]

Description

A single blocked/runaway RockConsumer on the shared in-memory rock-start-task-queue silently drops ALL start-task messages (registration confirmations, etc.) with no record, no exception, no log

Rock version: McKinley 18.3 (18.3.5)
Runtime: .NET Framework 4.8 (CLR v4.8.9325.00), IIS / w3wp
Bus transport: In-Memory (MassTransit) — Rock/Bus/Transport/Component/InMemory.cs
Severity: High — silent, unrecoverable loss of transactional email (event registration confirmations, receipts, etc.)

---

Summary

Rock sends several important messages as fire-and-forget publishes onto the in-memory MassTransit bus, all sharing a single queue, rock-start-task-queue (TTL 300s). Consumers (RockConsumer<TConsumer,TMessage>) run on the in-memory receive endpoint with the MassTransit defaults (no explicit concurrency, prefetch, or per-message timeout — see InMemory.cs).

If one consumer invocation blocks or runs away (e.g. a long/blocking plugin or HTTP call with no timeout, or a CPU-bound loop), the receive endpoint stops draining the queue. Every subsequent message on rock-start-task-queue then expires at the 300-second TTL and is discarded. Because:

1. the original publish is fire-and-forget (_ = RockMessageBus.SendAsync(...)), the publisher never observes the failure, and
2. TTL expiry / skipped delivery is not logged and raises no exception,

the result is completely silent loss. For registration confirmations specifically, the Communication record is created inside the consumer, so it is never created at all — which also means the periodic Send Communications job has nothing to retry. The only "fix" is an app-domain restart, which creates a fresh bus. Messages lost before the restart never arrive.

This turns any single misbehaving consumer into a silent, site-wide outage of all bus-delivered email until someone happens to restart the app.

---

Impact

• Event registration confirmation emails intermittently never send, with no error surfaced to the registrant, the admin, or the logs.
• The same queue also carries UpdateUserLastActivity, ProcessWorkflowTrigger, and other Rock.Tasks.* BusStartedTask messages — all of these stop being processed together.
• No Communication row is created, so the Send Communications job cannot recover the lost email.
• No exception, no entry in Rock.log / RockApplication.csv. The failure is invisible until a restart, after which new messages work again — making it extremely hard to diagnose.

---

Root cause / mechanism

The in-memory receive endpoint for rock-start-task-queue dispatches messages through a single delivery path with no bound on how long a consumer may run and no isolation between message types. A single blocked or looping Consume(...) therefore stalls the whole endpoint. Queued messages behind it age out at the 300s TTL and are dropped without a trace.

Confirmed by memory-dump analysis of a live wedged w3wp (two dumps 7 minutes apart show the identical stack — the delivery thread never advances):

MassTransit.InMemoryTransport.InMemoryReceiveTransport+ReceiveTransportAgent...Deliver
  MassTransit.Transports.ConsumerAgent`1.Dispatch
  MassTransit.Transports.ReceivePipeDispatcher.Dispatch
  ... (DeadLetter / Rescue / Deserialize / MessageType filters) ...
  MassTransit.Consumer.DelegateConsumerFactory`1.Send
  Rock.Bus.Consumer.RockConsumer`2.Consume(ConsumeContext`1)
  Rock.Tasks.ProcessWorkflowTrigger.Execute(Message)      <-- the single message being processed
  Rock.Model.WorkflowService.Process(...)
  Rock.Model.Workflow.ProcessActivities(...)
  Rock.Model.Workflow.get_ActiveActivities()              <-- never returns / re-evaluated forever

While this thread was pinned, the in-memory queue stopped consuming entirely: over one 7-minute window a published ProcessSendRegistrationConfirmation produced no Communication row and expired silently, and published UpdateUserLastActivity messages were likewise never consumed.

Note on the trigger in our environment: the specific consumer that hung was a misconfigured workflow on our side (an infinite activity-activation loop)

---

Suggested fixes / mitigations (any subset would help)

1. Per-message consume timeout on the receive endpoint so a single stuck consumer faults its own message instead of blocking the endpoint forever.
2. Never silently drop. Log (and ideally surface via the bus/health UI) whenever a start-task message faults or expires at TTL. Silent TTL expiry with no diagnostic is the core reason this is so hard to find.

---

Evidence appendix (controlled experiment)

Rock 18.3.5, in-memory bus, Mailgun HTTP medium (proven working — delivers in ~5s when the message is actually consumed).

Time (local)	Event	Published (Rock.log)	Consumed	Result
01:42:14	Resend confirmation, reg 87829	✓ ProcessSendRegistrationConfirmation	yes	Communication created 01:42:50, email delivered
~01:43:50	New registration instance created	✓ (workflow notification)	yes	Communication created (queue still healthy)
~01:43:50	Misconfigured workflow enters infinite loop on the delivery thread	—	—	consumer now pinned
01:44:09	New registration submitted, reg 87830	✓ ProcessSendRegistrationConfirmation	no	No Communication, no email, no error — expired at TTL
01:44–01:58	Queue dead for 14+ min	further publishes	no	all silently dropped
02:03	App pool recycled	—	—	fresh bus; new sends work again; lost messages never arrived

Dump analysis: the runaway consumer's in-memory List<WorkflowActivity> grew from 73,792 to 111,563 entries between the two dumps (~90/sec) while the delivery thread never left Workflow.get_ActiveActivities, and MAX(Communication.Id) did not advance for the entire window.

Actual Behavior

Messages get dropped without logging.

Expected Behavior

Never silently drop. Log (and ideally surface via the bus/health UI) whenever a start-task message faults or expires at TTL. Silent TTL expiry with no diagnostic is the core reason this is so hard to find.

Steps to Reproduce

1. Register a Rock.Tasks.* consumer (or trigger an existing one) whose Execute/Consume blocks indefinitely or loops (e.g. a workflow trigger looking for a registration to be saved, firing a workflow that instead of completing the workflow activates the complete workflow activity).
2. Cause that message to be published to rock-start-task-queue.
3. Submit an event registration for that event.
4. Observed: the delivery thread stays in the first consumer; the new messages never process; after 300s they vanish. No Communication record, no exception, no log line. The registrant never receives the confirmation.
5. Restart the app domain → bus is recreated → new registrations work again; previously lost messages never arrive.

Issue Confirmation

• Perform a search on the Github Issues to see if your bug is already reported.
• Reproduced the problem on a fresh install or on the demo site.

Rock Version

v18.3

Client Culture Setting

en-US

[MatthewKyle22]

Misconfigured workflow that caused the issue (Attached) - fired from a workflow trigger as per photo below.

Regisration Instance Created_202607121344.json