Octopus can crash if SQL is unavailable or the server is very busy when attempting to marking a task as complete

[vanessalove]

Customer has reported the Octopus server crashing. From the logs I see a lot of:

2015-08-04 11:58:29.5231     47 ERROR  Failed to append the following log entry:
CorrelationId: ServerTasks-728
Message: Delete release 4.0.36
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(IsolationLevel isolationLevel) in y:\work\refs\heads\master\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 60
   at Octopus.Server.Orchestration.Logging.Processors.MaskSensitiveValues.Initialize(IRelationalStore store) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\Processors\MaskSensitiveValues.cs:line 25
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.BuildChain(String id) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 109
   at Octopus.Server.Orchestration.Logging.Processors.Caching.SynchronizedCache`1.GetOrCreate(String id, Func`1 factory) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\Processors\Caching\SynchronizedCache.cs:line 53
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.ProcessLogEntry(String correlationId, ActivityLogEntry entry) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 80
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.SafelyAppend(String correlationId, ActivityLogEntry entry) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 62
2015-08-04 11:58:37.9293     12 ERROR  An unexpected error occurred while attempting to retrieve and execute a task: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(IsolationLevel isolationLevel) in y:\work\refs\heads\master\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 60
   at Octopus.Server.Orchestration.TaskQueue.PollForMoreWorkOnBackgroundThread(Object state) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\TaskQueue.cs:line 57
2015-08-04 12:36:19.3195      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.5244      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.5944      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.6594      6  INFO  The Octopus server is starting: Initializing database and performing migrations...

Before the service finally crashes.

Source: https://secure.helpscout.net/conversation/109846996/588/
(see Octopus.Server.0.txt from 2015-08-04 11:53:43.2843 )

[PaulStovell]

Should be fixed in 3.0.11

[PaulStovell]

Release Note: Prevent Octopus from crashing when marking a task as complete and the database is unavailable

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.