Transaction Queue Backup Issue

[tikool17]

Description

Description

On 18.2, our standard transaction queue intermittently stops processing and queued transactions build up until the IIS app pool is restarted. For us this is happening about once a week for about 2 months and we've tried to debug ourselves for many hours with no success. The main fallout for us is if it's stuck and we don't drain the queue before the app pool recycles itself overnight, those transactions are lost. [Disclosure: this issue writeup is assisted by ai, hopefully it's helpful]

When this happens:

• queued transaction counts continue to rise
• Drain Queue in the UI will process existing items, but new ones continue to pile up
• web interactions, workflows, etc stop being written while the queue is stuck
• draining the queue and then restarting the IIS app pool is the only fix we've found

Backed up queue:

Across multiple incidents, the pattern is:

1. an upstream timeout/failure occurs first
2. Rock logs RockQueue.Drain() exceptions
3. the queue stops processing until app pool restart

In several incidents, we observed one or more SQL/query timeouts before the later RockQueue.Drain() exceptions and queue-stuck behavior. We cannot confirm that every timeout is part of the same causal chain, but the pattern has repeated enough that it seems relevant.

These are the timeout sources we’ve seen so far, stated compactly:
Room Management reservation-change workflow
com.bemaservices.RoomManagement.Transactions.ReservationChangeTransaction.Execute() → workflow launch → Run Lava action → {% sql %} query timeout.
Communication Entry Wizard test send
Rock.Blocks.Communication.CommunicationEntryWizard.SendTestCommunication(...) → SaveChanges() timeout / SQL execution timeout.
Communications page grid load
Rock.Blocks.Communication.CommunicationList.GetGridData() → SQL read/query timeout while loading the staff Communications page grid.
Generic SQL timeout / wait timeout
Repeated Execution Timeout Expired and The wait operation timed out entries that appear before the later queue-drain exceptions.

Then after those, the recurring secondary exception pattern is:
Queue-drain/logging failure
RockQueue.Drain() → ToJson() → Person.ConnectionStatusValue → disposed ObjectContext.

Questions

1. Is there still an unhandled queue-drain/logging failure in 18.2 that can leave the standard transaction queue effectively stuck after upstream failures?
2. Can the ToJson() / queue logging path be hardened so it cannot mask the original exception with lazy-loaded/disposed entity errors?

Actual Behavior

Actual Behavior

• upstream timeout/failure occurs
• later RockQueue.Drain() throws secondary exceptions
• queue stops processing until IIS app pool restart

Exceptions below around the time of the last backup issue:

`

149106	4/21/2026 10:20 AM	The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.	at System.Data.Entity.Core.Objects.ObjectContext.get_Connection() at System.Data.Entity.Core.Objects.ObjectQuery1.GetResults(Nullable1 forMergeOption) at System.Data.Entity.Core.Objects.DataClasses.EntityReference1.Load(MergeOption mergeOption) at System.Data.Entity.Core.Objects.DataClasses.RelatedEnd.DeferredLoad() at System.Data.Entity.Core.Objects.Internal.LazyLoadBehavior.LoadProperty[TItem](TItem propertyValue, String relationshipName, String targetRoleName, Boolean mustBeNull, Object wrapperObject) at System.Data.Entity.Core.Objects.Internal.LazyLoadBehavior.<>c__DisplayClass0_02.b__2(TProxy proxy, TItem item) at System.Data.Entity.DynamicProxies.Person_[redacted].get_ConnectionStatusValue() at GetConnectionStatusValue(Object ) at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue(Object target)
149105	4/21/2026 10:20 AM	Error getting value from 'ConnectionStatusValue' on 'System.Data.Entity.DynamicProxies.Person_DA2FD4CD31D138D8D8E82E5FE88EE098EE5AB944A95EFDA6B3FF301DE647842B'.	at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue(Object target) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Rock.JsonExtensions.ToJson(Object obj) at Rock.Transactions.RockQueue.Drain(Action`1 errorHandler) in \Rock\Transactions\RockQueue.cs:line 242 at RockWeb.Global.DrainTransactionQueue() at System.Threading.Tasks.Task.Execute()
149104	4/21/2026 10:20 AM	A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.
149103	4/21/2026 10:20 AM	Unobserved Task Exception
149102	4/21/2026 10:12 AM	The wait operation timed out
149101	4/21/2026 10:12 AM	Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.[: Line 0]	at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func3 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed) at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryInternal[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__1() at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__0() at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryReliably[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQuery[TElement](String commandText, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Internal.LazyEnumerator1.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Rock.Blocks.Communication.CommunicationList.GetGridData()
149100	4/21/2026 10:12 AM	The wait operation timed out
149099	4/21/2026 10:12 AM	Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.[: Line 0]	at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func3 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed) at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryInternal[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__1() at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__0() at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryReliably[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQuery[TElement](String commandText, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Internal.LazyEnumerator1.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Rock.Blocks.Communication.CommunicationList.GetGridData()
149098	4/21/2026 10:12 AM	The wait operation timed out
149097	4/21/2026 10:12 AM	Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.[: Line 0]	at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func3 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed) at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryInternal[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__1() at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass186_01.<ExecuteStoreQueryReliably>b__0() at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQueryReliably[TElement](String commandText, String entitySetName, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Core.Objects.ObjectContext.ExecuteStoreQuery[TElement](String commandText, ExecutionOptions executionOptions, Object[] parameters) at System.Data.Entity.Internal.LazyEnumerator1.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Rock.Blocks.Communication.CommunicationList.GetGridData()
`

Expected Behavior

Expected Behavior

• A failing queued transaction should not leave the standard queue effectively wedged
• Queue-drain error logging should not throw a second exception while trying to serialize/log the failing transaction

Steps to Reproduce

We're not sure the root cause. Intermittent issues on our end on different days and times.

Issue Confirmation

• Perform a search on the Github Issues to see if your bug is already reported.
• Reproduced the problem on a fresh install or on the demo site.

Rock Version

18.2

Client Culture Setting

en-US

[cabal95]

@tikool17 We have added some additional safety logic around the draining of the transaction queue. This should prevent the queue from getting stuck. It should now correctly log the original error. However, just to note, because of where these errors occur, they do not get logged to the system exception table, since the database may not be operational. Instead they get logged to the Rock Logger system under the category of Rock.Transaction.