GetAllInstancesAsync thown Exception when use OrchestrationQuery

[kikutano]

Hello, when I try to use the GetAllInstancesAsync() Method:

var runtimeStatus = new List<OrchestrationRuntimeStatus>
{
        OrchestrationRuntimeStatus.Completed
 };

var instances = client.GetAllInstancesAsync(
       new OrchestrationQuery
       {
           Statuses = runtimeStatus,
           ContinuationToken = continuationToken,
       });

throws an Exception:

Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")

If i try to get all Instances without filters, works fine.

var instances = client.GetAllInstancesAsync();

    bool any = await instances
        .Where(
            x => x.RuntimeStatus == OrchestrationRuntimeStatus.Running ||
            x.RuntimeStatus == OrchestrationRuntimeStatus.Pending)
        .AnyAsync();

Any Idea?

[bachuv]

Hi @kikutano , thank you for opening this issue. I used the code that you provided to try and reproduce the issue and it's working for me. I don't see an exception. If possible, can you attach a zip file of a minimal code repro so we can locally debug the issue on our end? Thanks!

[Thundarr1974]

@bachuv We have run into this issue as well, although I don't know if it's the exact same error as the OP. We are encountering an RpcException but it can only be reproduced under a specific scenario:

1. Function app has at least 100 (or more) instances in Orchestration state, depending on the average size of data.
2. OrchestrationQuery "PageSize" is set to 20 or higher.

If we clear the orchestration state, or reduce the PageSize down to 10, then the error does not occur. Note that we have "FetchInputsAndOutputs" set to false and it does not prevent the exception from occurring.

Query

var query = new OrchestrationQuery
{
  InstanceIdPrefix = jobInfo.JobName,
  CreatedFrom = DateTimeOffset.UtcNow.Subtract(jobInfo.CachableDuration!.Value),
  FetchInputsAndOutputs = false,
  PageSize = 20                  
  Statuses = [OrchestrationRuntimeStatus.Completed]
};

Stack Trace

Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.")
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.<>c__DisplayClass18_0.<<GetAllInstancesAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+MoveNext()
   at Microsoft.DurableTask.Pageable.FuncAsyncPag
eable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Linq.AsyncEnumerable.TryGetFirst[TSource](IAsyncEnumerable`1 source, Func`2 predicate, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 147
   at System.Linq.AsyncEnumerable.TryGetFirst[TSource](IAsyncEnumerable`1 source, Func`2 predicate, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefau
lt.cs:line 147
   at System.Linq.AsyncEnumerable.<FirstOrDefaultAsync>g__Core|92_0[TSource](IAsyncEnumerable`1 source, Func`2 predicate, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 56
   at ******.<>c__DisplayClass9_1`2.<<FindJobResultAsync>b__1>d.MoveNext() in C:\*****.cs:line 199
--- End of stack trace from previous location ---
   at ****.InvokeDurableClient[T](Func`2 func) in C:\****.cs:line 346

[GregSam]

We have observed same Status(StatusCode="Unknown", Detail="Exception was thrown by handler.") issue.

We have found that this was caused by corrupted instance entry in the taskHub instances table.
Some Failed instances had missing properties, like CreatedTime.
For our use we can query with filter for Running and Pending, those seem to have all properties intact.
It looks like there is a serialization issue when entry do not match expected object?
We are thinking of implementing sanitization of instance entries when this error occur.
Why instances gets corrupted, dont know.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.