CallEntityAsync hangs indefinitely when targeting an unregistered entity type

[kshyju]

When an orchestration calls CallEntityAsync targeting an entity type that is not registered with any worker, the call hangs indefinitely instead of returning an error.

This is a breaking behavioral change. With SDK 1.18.0, the same call correctly failed with EntityOperationFailedException, allowing the orchestration to handle the error. With SDK 1.22.0, the entity message is sent to the scheduler but never dispatched to the worker — the orchestration waits forever with no error, no timeout, and no way to recover.

Environment

Component	Version
Microsoft.DurableTask.Worker.AzureManaged	1.22.0 (broken), 1.18.0 (works)
Microsoft.DurableTask.Client.AzureManaged	1.22.0 (broken), 1.18.0 (works)
DTS Emulator	mcr.microsoft.com/dts/dts-emulator:latest
.NET	10.0

Steps to reproduce

1. Start the DTS emulator:

   docker run -d --name dts-emulator -p 8080:8080 -e DTS_USE_DYNAMIC_TASK_HUBS=true mcr.microsoft.com/dts/dts-emulator:latest

2. Clone and run the repro:

   git clone https://github.com/kshyju/dts-entity-repro && cd dts-entity-repro && dotnet run

Expected behavior (works with SDK 1.18.0)

The orchestration completes. CallEntityAsync throws EntityOperationFailedException, the orchestration catches it, and completes successfully:

DurableTask SDK version: 1.18.0.0
Task hub: repro-b806ec

Starting orchestration that calls an UNREGISTERED entity type...
Waiting for orchestration c3ccc2e2... to complete (30s timeout)...
Orchestration completed: Status=Completed

Actual behavior (SDK 1.22.0)

The orchestration hangs indefinitely. The SendEntityMessage action is sent to the scheduler, but the scheduler never dispatches it to the worker:

DurableTask SDK version: 1.22.0.0
Task hub: repro-cb77ae

Starting orchestration that calls an UNREGISTERED entity type...
Waiting for orchestration 0b3355b8... to complete (30s timeout)...

BUG: Orchestration TIMED OUT after 30s.
The entity message to 'UnregisteredEntity' was never dispatched by the scheduler.
Expected the orchestration to fail with an error, not hang.

[github-actions]

Issue Triage

Classification: triage/needs-human-verification

Reasoning:
This is a well-documented regression between SDK 1.18.0 and 1.22.0. Investigation reveals two potential contributing factors:

1. WorkItemFilters (PR Introduce WorkItemFilters into worker flow #616): SDK 1.22.0 introduced DurableTaskWorkerWorkItemFilters which are auto-generated from the DurableTaskRegistry. Only registered entity types are included in the filters sent to the scheduler. When an orchestration calls CallEntityAsync for an unregistered entity, the scheduler has no matching worker filter and the message is never dispatched — causing the orchestration to hang at WaitForExternalEvent.

2. EntityRequest (V1) path missing operationInfos: In GrpcDurableTaskWorker.Processor.cs (line 304), the V1 EntityRequest path calls OnRunEntityBatchAsync without extracting operationInfos (routing info). Even if the message were dispatched, the error response for an unregistered entity would lack responseDestination, preventing the scheduler from routing the error back to the waiting orchestration. The V2 EntityRequestV2 path (lines 307-320) correctly extracts and passes operationInfos.

The fix likely requires coordination between the SDK and the DTS scheduler/emulator. The scheduler should return an error when no worker matches an entity message, and the V1 entity path should also preserve routing information.

Automated fix candidate: No

• Root cause spans SDK and DTS scheduler/emulator interaction
• Requires domain expertise to determine the correct fix location (SDK, scheduler, or both)
• Potential for unintended side effects in multi-worker entity routing scenarios

[cgillum]

This is most likely due to #616 combined with the new work item filter feature that was added to DTS. We may need to change the default SDK behavior to not opt-into work item filters as this would be considered a breaking change.

[YunchuWang]

Fix submitted in PR #669.

Root cause: PR #616 introduced work item filters that are auto-generated from the DurableTaskRegistry and sent to the DTS scheduler. The scheduler only dispatches work items matching those filters. When an orchestration calls CallEntityAsync targeting an unregistered entity type, no worker has a matching filter — the entity message sits in the base queue that no filtered worker reads, causing the orchestration to hang indefinitely.

Fix: Changed work item filters from auto opt-in to explicit opt-in. By default, no filters are sent and the worker processes all work items (pre-1.22.0 behavior). Users who want filtering can explicitly opt-in by calling builder.UseWorkItemFilters().

E2E test results against real DTS scheduler:

=== Work Item Filter Regression Test (Issue #668) ===

[TEST] CallEntityAsync targeting unregistered entity fails with error
  Orchestration completed with output: OK: Got EntityOperationFailedException -
  EntityTaskNotFound: No entity task named 'unregisteredentity' was found.
  RESULT: PASS

[TEST] CallEntityAsync targeting registered entity succeeds
  Orchestration completed with output: Counter value: 0
  RESULT: PASS

[TEST] CallActivityAsync targeting unregistered activity fails with error
  Orchestration failed as expected: OrchestratorTaskNotFound
  RESULT: PASS

=== Results: 3 passed, 0 failed ===

@cgillum @halspang — requesting review.