Grpc Error calling WaitForInstanceCompletionAsync in dotnet-isolated

[rob-baily]

I don't know that it is exactly the same issue as #3227 but the error message with Grpc.Core.RpcException is the same and changing the storage mechanism had no effect. For me the code below is taken from the template and I added the await client.WaitForInstanceCompletionAsync call which is where the error is triggered. Changing storage does not help at all. If anyone has any ideas or if anyone from MS can triage that would be great. It is really easy to reproduce since it is basically the MS template with one line added. This is using .NET 7.0 in an isolated process. I upgraded to the latest packages so here are my versions:

    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.13.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.9.0" />

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace DurableFunctionExample
{
    public static class Function
    {
        [Function(nameof(Function))]
        public static async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] TaskOrchestrationContext context)
        {
            ILogger logger = context.CreateReplaySafeLogger(nameof(Function));
            logger.LogInformation("Saying hello.");
            var outputs = new List<string>();

            // Replace name and input with values relevant for your Durable Functions Activity
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Tokyo"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Seattle"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "London"));

            // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
            return outputs;
        }

        [Function(nameof(SayHello))]
        public static string SayHello([ActivityTrigger] string name, FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("SayHello");
            logger.LogInformation("Saying hello to {name}.", name);
            return $"Hello {name}!";
        }

        [Function("Function_HttpStart")]
        public static async Task<HttpResponseData> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
            [DurableClient] DurableTaskClient client,
            FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("Function_HttpStart");

            // Function input comes from the request content.
            string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
                nameof(Function));

            logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

            var metaData = await client.WaitForInstanceCompletionAsync(instanceId);
            // Returns an HTTP 202 response with an instance management payload.
            // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
            return client.CreateCheckStatusResponse(req, instanceId);
        }
    }
}

The full stack is:

[2023-04-12T17:49:49.217Z] Function 'Function_HttpStart', Invocation id 'ddbe2542-1404-45af-b96b-ec9762a28dd2': An exception was thrown by the invocation.
[2023-04-12T17:49:49.221Z] Result: Function 'Function_HttpStart', Invocation id 'ddbe2542-1404-45af-b96b-ec9762a28dd2': An exception was thrown by the invocation.
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler."))
[2023-04-12T17:49:49.222Z]  ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-04-12T17:49:49.224Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
[2023-04-12T17:49:49.225Z]    at DurableFunctionExample.Function.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\RobBaily\source\repos\DurableFunctionExample\Function.cs:line 50
[2023-04-12T17:49:49.227Z]    --- End of inner exception stack trace ---
[2023-04-12T17:49:49.228Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.230Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.232Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.234Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.236Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.237Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.237Z] Executed 'Functions.Function_HttpStart' (Failed, Id=ddbe2542-1404-45af-b96b-ec9762a28dd2, Duration=360ms)
[2023-04-12T17:49:49.240Z] System.Private.CoreLib: Exception while executing function: Functions.Function_HttpStart. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler."))
[2023-04-12T17:49:49.238Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.241Z]  ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-04-12T17:49:49.243Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
[2023-04-12T17:49:49.242Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.247Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.245Z]    at DurableFunctionExample.Function.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\RobBaily\source\repos\DurableFunctionExample\Function.cs:line 50
[2023-04-12T17:49:49.248Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.251Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.253Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.256Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.250Z]    --- End of inner exception stack trace ---
[2023-04-12T17:49:49.257Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.259Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.260Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.262Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.263Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.266Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.267Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.269Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.270Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.271Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.272Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.267Z] Executing 'Functions.Function' (Reason='(null)', Id=8c59ab82-641b-45e7-b164-d8dbb023820b)
[2023-04-12T17:49:49.274Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.277Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.264Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.278Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77.
[2023-04-12T17:49:49.280Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.282Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.284Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.285Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.286Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.288Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.289Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.291Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.293Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.294Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2023-04-12T17:49:49.295Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request, WorkerOptions workerOptions) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 84
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.296Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.297Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.298Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.300Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.301Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.302Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.304Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.306Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.307Z] Saying hello.
[2023-04-12T17:49:49.308Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.310Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.311Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.312Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2023-04-12T17:49:49.313Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request, WorkerOptions workerOptions) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 84.

[rob-baily]

Have also confirmed the same behavior using 6.0.

Here are the versions in that instance:

    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.10.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.7.0" />

[rob-baily]

Updated the subject line as issue occurs in 6.0 as well. Also confirm it happens when using a published version on Azure.

[torepaulsson]

This has happened to me as well, same log message and no way of knowing what is wrong!

[todosrc]

i have same issue

[AdiThakker]

I am facing the same issue as well, when invoking WaitForInstanceCompletionAsync. Following is sample code.

Error returned is " Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")

I can see that the Azurite Tables HubNameInstances table shows Orchestration Status as Running

[Schmaga]

We also suffer from the same Grpc Error. We receive essentially the same stack trace, when calling WaitForCompletionAsync and also PurgeInstancesAsync.

[todosrc]

We also suffer from the same Grpc Error. We receive essentially the same stack trace, when calling WaitForCompletionAsync and also PurgeInstancesAsync.

it need register durable/grpc onstartup.cs

services.AddDurableTaskClient(builder =>
{
// Configure options for this builder. Can be omitted if no options customization is needed.
builder.Configure(opt => { });
builder.UseGrpc(); // multiple overloads available for providing gRPC information

        // AddDurableTaskClient allows for multiple named clients by passing in a name as the first argument.
        // When using a non-default named client, you will need to make this call below to have the
        // DurableTaskClient added directly to the DI container. Otherwise IDurableTaskClientProvider must be used
        // to retrieve DurableTaskClients by name from the DI container. In this case, we are using the default
        // name, so the line below is NOT required as it was already called for us.
        builder.RegisterDirectly();
    });

[Schmaga]

it need register durable/grpc onstartup.cs

services.AddDurableTaskClient(builder => { // Configure options for this builder. Can be omitted if no options customization is needed. builder.Configure(opt => { }); builder.UseGrpc(); // multiple overloads available for providing gRPC information

I tried that because I saw that in the other issue. It did not help. Also, there is absolutely no documentation on that approach anywhere. Not in the migration guides, nor other dotnet-isolated Function resources that I could find.