Under what circumstance does DTF SQL provider throws Duplicate execution error

[jundayin]

Would like to here input from DTFx expert
 
We notice the DTF throws error like Duplicate execution of 'orchestration' was detected! This means two workers or threads tried to process the same work item simultaneously. The result of this execution will be discarded on our customer machine which has SQL Server 2019 installed and the orchestration just stuck. Occasionally, there is also database connection timeout issue. However, we never encounter this problem on Azure SQL even with the lowest hardware setup.

This doesn't happen often, but it gave us headache on investigating the root cause. We were able to resolve the problem once when customer changed their disk to a more performant one. However, another customer with SSD disk also encountered this error which doesn't make sense to us.

It would be great if you as the owner of the framework could give us some insights regarding the duplicated execution
 
Current DT version: 1.0.0-rc

[cgillum]

Can you clarify what 1.0.0-rc refers to? We don't have a Durable Task version of that number. Perhaps you're referring to a nuget package version, such as Microsoft.DurableTask.SqlServer?

Assuming so, that version is quite old and there may be bugs. It is recommended to upgrade to the latest version, such as 1.2.3, which also includes important security fixes.

[jundayin]

Thanks @cgillum . Yes, I was referring the version of SQL Provider nuget package.

I understand it is rather old and we are planning to upgrade to latest version this year, but still we would like to understand the problem better to solve our customer issue.

Do you believe setting the max concurrent activity and orchestration to 1 can resolve the issue?

[jundayin]

@cgillum Any thought? It would be appreciated if you can provide us some insights of the issue here as we need to unblock out customer installation.

Something we are confused currently:

1. Under what circumstance there will be duplicated execution?
2. Do you believe set the max concurrent activity and orchestration can resolve the issue?
3. If not, how do we ensure the problem does not happen on the latest Microsoft.DurableTask.SqlServer

To provide more clarity. This is the message emitted by DTF MSSQL

2024-04-10 12:46:17.6562|WARN|DurableTask.SqlServer|3d884a6a9fb5498d8c9744f601be1b5f:1: Duplicate execution of 'orchestration' was detected! This means two workers or threads tried to process the same work item simultaneously. The result of this execution will be discarded.
2024-04-10 12:46:17.6577|WARN|DurableTask.SqlServer|3d884a6a9fb5498d8c9744f601be1b5f:1: Duplicate execution of 'orchestration' was detected! This means two workers or threads tried to process the same work item simultaneously. The result of this execution will be discarded.

And then the DT throws this exception under for the same orchestration

2024-04-10 12:46:17.9671|ERROR|DurableTask.Core|TaskActivityDispatcher-c0289b205b9648d09c615d143bf78c4c-0: Unhandled exception with work item '3d884a6a9fb5498d8c9744f601be1b5f:1:0000000000000003': Microsoft.Data.SqlClient.SqlException (0x80131904): The target instance is not running. It may have already completed (in which case this execution can be considered a duplicate) or been terminated. Any results of this task activity execution will be discarded."}
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)"}
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)"}
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)"}
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)"}
   at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)"}
   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)"}
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)"}
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)"}
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)"}
--- End of stack trace from previous location ---"}
   at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 500"}
   at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 507"}
   at DurableTask.SqlServer.SqlUtils.ExecuteSprocAndTraceAsync[T](DbCommand command, LogHelper traceHelper, String instanceId, Func`2 executor) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 449"}
   at DurableTask.SqlServer.SqlOrchestrationService.CompleteTaskActivityWorkItemAsync(TaskActivityWorkItem workItem, TaskMessage responseMessage) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 400"}
   at DurableTask.Core.TaskActivityDispatcher.OnProcessWorkItemAsync(TaskActivityWorkItem workItem) in /_/src/DurableTask.Core/TaskActivityDispatcher.cs:line 266"}
   at DurableTask.Core.TaskActivityDispatcher.OnProcessWorkItemAsync(TaskActivityWorkItem workItem) in /_/src/DurableTask.Core/TaskActivityDispatcher.cs:line 266"}
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 459"}
ClientConnectionId:8dd9266e-3f22-45f7-8455-0f3b15474870"}
Error Number:50003,State:1,Class:16"}
Backing off for 1 seconds until 5 successful operations"}

[cgillum]

@jundayin your customer is using a prerelease version of this package, which we don't intend to support.

The cause of these errors is likely related to the worker being overloaded but could also be related to bugs that have been fixed in newer versions of the package. I can't guarantee that you won't see this in a future version.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.