Multi-instance query support

[cgillum]

Requirement

Orchestration clients should be able to issue filtered queries to the state store for orchestration instances, as described here.

Basic Instructions

There are quite a few steps required to get this working end-to-end involving updating the Java SDK, updating the gRPC contract, updating the sidecar, and updating the Durable Task Framework (DTFx). The good news is that the changes should all be fairly simple. Here are some instructions for how this can be done:

Durable Task Framework updates

The https://github.com/Azure/durabletask repo is where DTFx lives. It defines the core data types and backend storage providers that are common across all architectures. Unfortunately, there is no common abstraction for multi-instance query support. We will need to create this abstraction and then update the existing storage providers to support it. This shouldn't require much work but will require several PRs across several GitHub repos.

1. Define a new public IOrchestrationServiceQueryClient interface. It can define a single method for querying for multiple orchestration instances. The design needs to be compatible with the existing methods mentioned below so that each storage provider can successfully implement it. The methods below should hopefully all be compatible with each other.

2. Update the AzureStorageOrchestrationService class in Azure/durabletask to implement this new interface and map it to the existing GetOrchestrationStateAsync method.

3. Update the SqlOrchestrationService class in microsoft/durabletask-mssql to implement this new interface and map it to the existing GetManyOrchestrationsAsync method.

4. Update the NetheriteOrchestrationService class in microsoft/durabletask-netherite to implement this new interface and map it to the existing QueryOrchestrationStatesAsync method.

It's not strictly necessary for us to do this work for all backends at once. We can start with (1) and skip ahead to the next section so we can more quickly get to integration testing and validate the design. I suggest we open GitHub issues to separately track the different steps above.

Sidecar Updates

These changes are required for integration testing to be unblocked. This work will also benefit the .NET Isolated support for Durable Functions.

1. Update the InMemoryOrchestrationService class in the microsoft/durabletask-sidecar repo to implement this new interface. There is no existing method that implements similar functionality, so an implementation will need to be written from scratch. This is an in-memory storage provider so it's fine to just enumerate all the objects in InMemoryInstanceStore.store and return the objects that match the filter. This is the storage provider used by integration tests so this implementation should probably be the first one to work on.

2. Uncomment and finish implementing the QueryInstances gRPC definition. For local development, these changes can be done directly in the sub-module under the microsoft/durabletask-sidecar project. This API will need to match the API defined in the new IOrchestrationServiceQueryClient method mentioned above.

3. Using the updated gRPC definition, add a new overload method to the sidecar's TaskHubGrpcServer class. This method will take the IOrchestrationServiceClient object (which should already be accessible) and try to cast it into IOrchestrationServiceQueryClient. If the cast succeeds, then invoke the method. Otherwise, throw a new NotSupportedException with a helpful error message.

Java SDK Updates (finally)

These changes are required specifically for Java.

1. In the Java SDK, add a new method to DurableTaskClient with a matching method signature for the new multi-instance query gRPC API. The implementation should simply call the new gRPC method that was defined in a previous step.

2. Several test permutations can be written for integration testing. My suggestion is to take a look at the MultiInstanceQueries tests in the SQL backend project. It has pretty good coverage with a relatively simple test definition. The test could be broken up into multiple tests or kept as one. Either way is fine.

So there's a decent a mount of infrastructure work required for this work item, but it will also benefit the .NET Isolated work as well as any direct users of the Durable Task Framework (which there are many within Microsoft), so the impact is much larger than just our Java users.