Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Octopus can crash if SQL is unavailable or the server is very busy when attempting to marking a task as complete #1851

Closed
vanessalove opened this issue Aug 6, 2015 · 3 comments

Comments

@vanessalove
Copy link
Contributor

commented Aug 6, 2015

Customer has reported the Octopus server crashing. From the logs I see a lot of:

2015-08-04 11:58:29.5231     47 ERROR  Failed to append the following log entry:
CorrelationId: ServerTasks-728
Message: Delete release 4.0.36
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(IsolationLevel isolationLevel) in y:\work\refs\heads\master\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 60
   at Octopus.Server.Orchestration.Logging.Processors.MaskSensitiveValues.Initialize(IRelationalStore store) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\Processors\MaskSensitiveValues.cs:line 25
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.BuildChain(String id) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 109
   at Octopus.Server.Orchestration.Logging.Processors.Caching.SynchronizedCache`1.GetOrCreate(String id, Func`1 factory) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\Processors\Caching\SynchronizedCache.cs:line 53
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.ProcessLogEntry(String correlationId, ActivityLogEntry entry) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 80
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.SafelyAppend(String correlationId, ActivityLogEntry entry) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Logging\ServerLogStorage.cs:line 62
2015-08-04 11:58:37.9293     12 ERROR  An unexpected error occurred while attempting to retrieve and execute a task: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
   at Microsoft.WindowsAzure.Common.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Octopus.Core.RelationalStorage.RelationalStore.BeginTransaction(IsolationLevel isolationLevel) in y:\work\refs\heads\master\source\Octopus.Core\RelationalStorage\RelationalStore.cs:line 60
   at Octopus.Server.Orchestration.TaskQueue.PollForMoreWorkOnBackgroundThread(Object state) in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\TaskQueue.cs:line 57
2015-08-04 12:36:19.3195      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.5244      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.5944      6  INFO  Browse your Octopus server at: http://localhost:80/
2015-08-04 12:36:19.6594      6  INFO  The Octopus server is starting: Initializing database and performing migrations...

Before the service finally crashes.

Source: https://secure.helpscout.net/conversation/109846996/588/
(see Octopus.Server.0.txt from 2015-08-04 11:53:43.2843 )

@PaulStovell PaulStovell closed this Aug 6, 2015
@PaulStovell PaulStovell removed the ready label Aug 6, 2015
@PaulStovell PaulStovell changed the title Getting Octopus Service crashes on timeout expired. Octopus can crash if SQL is unavailable or the server is very busy when attempting to marking a task as complete Aug 6, 2015
@PaulStovell PaulStovell added this to the 3.0.11 milestone Aug 6, 2015
@PaulStovell

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

Should be fixed in 3.0.11

@PaulStovell

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

Release Note: Prevent Octopus from crashing when marking a task as complete and the database is unavailable

@PaulStovell PaulStovell self-assigned this Aug 6, 2015
@lock

This comment has been minimized.

Copy link

commented Nov 26, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.

@lock lock bot locked as resolved and limited conversation to collaborators Nov 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.