Durable entities in isolated mode fail on CallEntityAsync

[oleks9o6]

I'm trying to use durable entities in our project but I'm getting an error when calling an operation on the entity in an orchestrator.

I'm using .NET 8, isolated worker mode, Microsoft.Azure.Functions.Worker.Extensions.DurableTask version is 1.1.0.

That's the call from the orchestrator:

    var lastUpdated = await context.Entities.CallEntityAsync<DateTime?>(entityId, "Get");

the entity definition is:

    public class HistoricalLastTakenAt
    {
        public DateTime? Value { get; set; }

        public DateTime? Get() => Value;

        public void Set(DateTime value)
        {
            if (value > Value) Value = value;
        }

        [Function(nameof(HistoricalLastTakenAt))]
        public static Task RunEntityAsync([EntityTrigger] TaskEntityDispatcher dispatcher)
        {
            return dispatcher.DispatchAsync<HistoricalLastTakenAt>();
        }
    }

but it throws with a null reference

the stack trace is:

DurableTask.Core.dll!DurableTask.Core.Entities.OrchestrationEntityContext.AdjustOutgoingMessage(string instanceId, DurableTask.Core.Entities.EventFormat.RequestMessage requestMessage, System.DateTime? cappedTime, out string eventName) Line 311	C#
 	DurableTask.Core.dll!DurableTask.Core.Entities.OrchestrationEntityContext.EmitRequestMessage(DurableTask.Core.OrchestrationInstance target, string operationName, bool oneWay, System.Guid operationId, (System.DateTime Original, System.DateTime Capped)? scheduledTimeUtc, string input) Line 243	C#
 	Microsoft.DurableTask.Worker.dll!Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.SendOperationMessage(string instanceId, string operationName, object input, bool oneWay, System.DateTimeOffset? scheduledTime)	C#
 	Microsoft.DurableTask.Worker.dll!Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityInternalAsync(Microsoft.DurableTask.Entities.EntityInstanceId id, string operationName, object input)	C#
 	Microsoft.DurableTask.Worker.dll!Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityAsync<System.DateTime?>(Microsoft.DurableTask.Entities.EntityInstanceId id, string operationName, object input, Microsoft.DurableTask.Entities.CallEntityOptions options)	C#
 	Microsoft.Azure.Functions.Worker.Extensions.DurableTask.dll!Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrationContext.EntityFeature.CallEntityAsync<System.DateTime?>(Microsoft.DurableTask.Entities.EntityInstanceId id, string operationName, object input, Microsoft.DurableTask.Entities.CallEntityOptions options) Line 29	C#
 	Scalepoint.ForsiReportManager.dll!Scalepoint.ForsiReportManager.Functions.HistoricalReports.ProcessHistoricalReportsForTenant.Run(Microsoft.DurableTask.TaskOrchestrationContext context, string tenant, Microsoft.Azure.Functions.Worker.FunctionContext functionContext, System.Threading.CancellationToken cancellationToken) Line 32	C#
 	[Lightweight Function]	
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker<Scalepoint.ForsiReportManager.Functions.HistoricalReports.ProcessHistoricalReportsForTenant, object>.InvokeAsync(Scalepoint.ForsiReportManager.Functions.HistoricalReports.ProcessHistoricalReportsForTenant instance, object[] arguments) Line 22	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker<Scalepoint.ForsiReportManager.Functions.HistoricalReports.ProcessHistoricalReportsForTenant, object>.InvokeAsync(object instance, object[] arguments) Line 31	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 45	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.Pipeline.FunctionExecutionMiddleware.Invoke(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 20	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseFunctionExecutionMiddleware.AnonymousMethod__1_2(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 57	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(Microsoft.Azure.Functions.Worker.FunctionContext context, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next) Line 13	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseOutputBindingsMiddleware.AnonymousMethod__3(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 84	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseWhen.AnonymousMethod__1(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 206	C#
 	Scalepoint.ForsiReportManager.dll!Scalepoint.ForsiReportManager.Middleware.AuthorizationMiddleware.Invoke(Microsoft.Azure.Functions.Worker.FunctionContext context, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next) Line 48	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseMiddleware.AnonymousMethod__1(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 105	C#
 	Scalepoint.ForsiReportManager.dll!Scalepoint.ForsiReportManager.Middleware.AuthenticationMiddleware.Invoke(Microsoft.Azure.Functions.Worker.FunctionContext context, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next) Line 48	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseMiddleware.AnonymousMethod__1(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 105	C#
 	Microsoft.Azure.Functions.Worker.Extensions.DurableTask.dll!Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(Microsoft.Azure.Functions.Worker.FunctionContext functionContext, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next, Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrationContext orchestrationContext) Line 72	C#
 	Microsoft.Azure.Functions.Worker.Extensions.DurableTask.dll!Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(Microsoft.DurableTask.TaskOrchestrationContext context, object input) Line 51	C#
 	Microsoft.DurableTask.Worker.dll!Microsoft.DurableTask.Worker.Shims.TaskOrchestrationShim.Execute(DurableTask.Core.OrchestrationContext innerContext, string rawInput)	C#
 	DurableTask.Core.dll!DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(DurableTask.Core.History.HistoryEvent historyEvent) Line 211	C#
 	DurableTask.Core.dll!DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore.__ProcessEvents|12_0(System.Collections.Generic.IEnumerable<DurableTask.Core.History.HistoryEvent> events) Line 136	C#
 	DurableTask.Core.dll!DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(System.Collections.Generic.IEnumerable<DurableTask.Core.History.HistoryEvent> pastEvents, System.Collections.Generic.IEnumerable<DurableTask.Core.History.HistoryEvent> newEvents) Line 153	C#
 	DurableTask.Core.dll!DurableTask.Core.TaskOrchestrationExecutor.Execute() Line 93	C#
 	Microsoft.DurableTask.Worker.Grpc.dll!Microsoft.DurableTask.Worker.Grpc.GrpcOrchestrationRunner.LoadAndRun(string encodedOrchestratorRequest, Microsoft.DurableTask.ITaskOrchestrator implementation, System.IServiceProvider services)	C#
 	Microsoft.Azure.Functions.Worker.Extensions.DurableTask.dll!Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(Microsoft.Azure.Functions.Worker.FunctionContext context, Microsoft.Azure.Functions.Worker.BindingMetadata triggerBinding, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next) Line 58	C#
 	Microsoft.Azure.Functions.Worker.Extensions.DurableTask.dll!Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(Microsoft.Azure.Functions.Worker.FunctionContext functionContext, Microsoft.Azure.Functions.Worker.Middleware.FunctionExecutionDelegate next) Line 30	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.UseMiddleware.AnonymousMethod__1(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 105	C#
 	Microsoft.Azure.Functions.Worker.Core.dll!Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(Microsoft.Azure.Functions.Worker.FunctionContext context) Line 77	C#
 	Microsoft.Azure.Functions.Worker.Grpc.dll!Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(Microsoft.Azure.Functions.Worker.Grpc.Messages.InvocationRequest request) Line 88	C#
 	Microsoft.Azure.Functions.Worker.Grpc.dll!Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(Microsoft.Azure.Functions.Worker.Grpc.Messages.InvocationRequest request) Line 122	C#
 	Microsoft.Azure.Functions.Worker.Grpc.dll!Microsoft.Azure.Functions.Worker.GrpcWorker.ProcessRequestCoreAsync(Microsoft.Azure.Functions.Worker.Grpc.Messages.StreamingMessage request) Line 81	C#
 	Microsoft.Azure.Functions.Worker.Grpc.dll!Microsoft.Azure.Functions.Worker.GrpcWorker.Microsoft.Azure.Functions.Worker.Grpc.IMessageProcessor.ProcessMessageAsync.AnonymousMethod__0() Line 66	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.Task>.InnerInvoke()	C#
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread)	C#
 	System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()	C#
 	[Native to Managed Transition]	

[kshyju]

What version of Microsoft.Azure.Functions.Worker and Microsoft.Azure.Functions.Worker.Sdk packages are you using?

[oleks9o6]

What version of Microsoft.Azure.Functions.Worker and Microsoft.Azure.Functions.Worker.Sdk packages are you using?

Microsoft.Azure.Functions.Worker is 1.19.0, Microsoft.Azure.Functions.Worker.Sdk is 1.15.1.

[lilyjma]

HI @oleks9o6, are you still experiencing the problem? If so, could you provide a repro? Thanks!

[oleks9o6]

Hi @lilyjma,

Yes, the issue is still reproducible for me. I created a minimal repro example here: https://github.com/oleks9o6/durabletask-dotnet-issues-246. It is durable functions in isolated mode, and runs on .NET 8.0. It also uses MSSQL Server as a storage.
The simplified example uses the latest stable versions of nuget packages.

The exception from the above occurs when I'm trying to call the durable entity here: https://github.com/oleks9o6/durabletask-dotnet-issues-246/blob/main/src/FunctionApp1/FunctionApp1/Function1.cs#L54.

[sebastianburckhardt]

If I am not mistaken, mssql does not yet support isolated entities. However, we would expect there to be a better error message than just throwing a null reference exception.

[cgillum]

If I am not mistaken, mssql does not yet support isolated entities. However, we would expect there to be a better error message than just throwing a null reference exception.

@sebastianburckhardt is there a recommended fix that should be made in DurableTask.Core to throw a better exception if IEntityOrchestrationService is not implemented?

[sebastianburckhardt]

It appears to me that the reason why we get this exception (instead of a normal error message) is that the check that is meant to recognize and report this problem (see LocalGrpcListener.CheckEntitySupport) is not triggered because the code fails before even calling the grpc service, because it accesses a null entity parameter.