Prevent races between orchestration-invoker and user code

[davidmrdavid]

The problem:

Our code runs two threads, one for the user code, and another for the orchestration-invoker. The orchestration-invoker runs first, and it starts the user code thread via a BeginInvoke method, as shown below in line 69.

https://github.com/Azure/azure-functions-durable-powershell-private/blob/76c1801e72a033e8e588ceebc3b50f96bfe29cc9/src/DurableEngine/OrchestrationInvoker.cs#L59-L77

At this moment, and for a short amount of time, there's technically two threads racing against each other. As seen in the snippet above, the orchestration-invoker thread will immediately block itself by yielding to the user code thread by calling YieldToUserCodeThread in line 76.

The expectation is that the orchestration-invoker thread will block itself before the user code thread gets to block itself by calling YieldToInvokerThread in line 56 below.

https://github.com/Azure/azure-functions-durable-powershell-private/blob/76c1801e72a033e8e588ceebc3b50f96bfe29cc9/src/DurableEngine/Tasks/DurableTask.cs#L39-L56

That expectation (that the invoker thread yields first) is met in practice and it ensures that, most of the time, only 1 of the 2 threads is running. However, at the very beginning of replay, it is technically possible for the user code thread to yield first as the two threads execute concurrently for a short time.

This would be a problem because our logic depends on the orchestration-invoker thread yielding first.

This PR

This PR introduces a simple change to guarantee that our assumption is met. In the beginning of the Execute method of DurableTask, we now check for whether or not the WaitHandle for "userCodeThreadTurn" has been signaled. If it has, we continue running. If it hasn't been signaled, we block.

When the orchestration invoker yields, it already sets this handle, so this check is essentially the same as checking whether the orchestration invoker has already yielded once.