Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQL Deadlocks with CommandBatchMaxTimeout #1124

Closed
shorbachuk opened this issue Mar 2, 2018 · 5 comments
Closed

SQL Deadlocks with CommandBatchMaxTimeout #1124

shorbachuk opened this issue Mar 2, 2018 · 5 comments
Labels
a: sql-server t: bug
Milestone
Hangfire 1.6.18

Comments

@shorbachuk
Copy link

I am using the SqlServerStorageOptions.CommandBatchMaxTimeout documented here:

https://www.hangfire.io/blog/2017/09/20/hangfire-1.6.17.html

I have been getting deadlocks.

Originally posted here:
pieceofsummer/Hangfire.Console#57

`System.Data.SqlClient.SqlException
Transaction (Process ID 169) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

System.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 169) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at System.Data.SqlClient.SqlCommandSet.ExecuteNonQuery() at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery() at Hangfire.SqlServer.SqlServerConnection.<>c__DisplayClass14_0.<SetRangeInHash>b__0(DbConnection connection, DbTransaction transaction) at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass26_0.<UseTransaction>b__0(DbConnection connection, DbTransaction transaction) at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func2 func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](DbConnection dedicatedConnection, Func3 func, Nullable1 isolationLevel)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnection dedicatedConnection, Action2 action) at Hangfire.SqlServer.SqlServerConnection.SetRangeInHash(String key, IEnumerable1 keyValuePairs)
at Hangfire.Console.Storage.ConsoleStorage.InitConsole(ConsoleId consoleId)
at Hangfire.Console.Server.ConsoleServerFilter.OnPerforming(PerformingContext filterContext)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func1 continuation)
@odinserj odinserj added this to the Hangfire 1.6.18 milestone Mar 2, 2018
@plaisted
Copy link
Contributor

plaisted commented Mar 8, 2018

Any details on how to avoid this? Deployed to a new server today and am getting this constantly now. Same Hangfire version on another server works fine. Haven't dug into the bug but curious if there's a quick fix.

@shorbachuk
Copy link
Author

Are you using the CommandBatchMaxTimeout option?

@plaisted
Copy link
Contributor

plaisted commented Mar 9, 2018

I'm not, but the stack trace is identical. Perhaps the bug isn't related to CommandBatchMaxTimeout but just gets triggered by it in some cases.

odinserj added a commit that referenced this issue Mar 12, 2018
@odinserj
Use custom lock in SqlConnection.SetRangeInHash to prevent deadlocks
8b3fbc1 
There are a lot of indexes in the Hash table, and the clustered one is
simply based on an incremental id. Without using a lock as in
transactions, sooner or later this method will cause a deadlock. Without
using a lock as in transactions, sooner or later this method will cause
a deadlock.
Fixes #1124
@odinserj odinserj mentioned this issue Mar 12, 2018
Merged
@pieceofsummer
Copy link
Contributor

I could probably wrap SetRangeInHash into a transaction to make Console work with current version of Hangfire.

@shorbachuk
Copy link
Author

Is there a release planned for 1.6.18?

@odinserj odinserj mentioned this issue Mar 30, 2018
Merged
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
a: sql-server t: bug
Milestone
Hangfire 1.6.18
Development

No branches or pull requests
4 participants
@odinserj @pieceofsummer @plaisted @shorbachuk