feat: Provide Process Instance Search Request API in the Zeebe Client

[romansmirnov]

Description

Implements a search request API in the Zeebe Client so users can search process instances creating a query, for example

final SearchQueryResponse<ProcessInstance> response = client.newProcessInstanceQuery()
        .filter((f) -> f.variable((v) -> v.name("foo").eq(5)))
        .sort((s) -> s.endDate().asc())
        .page((p) -> p.size(20))
        .send()
        .join();

Related issues

closes #

[romansmirnov]

How can the Zeebe Client (upcoming Camunda Client) be used to execute a search request?

Basically, the API reflects that a search query consists of three properties:

1. filter: containing the entity-specific filter properties,
2. sort: containing the entity-specific sorting properties, and
3. page that describes the next page to retrieve (not specific to any entity - might not be correct in all cases like the identity entities).

The API offers a fluent builder to construct a (simple) search query. Additionally, static builders exist that help to construct complex queries and allow the reuse of defined filters.

Fluent Builder

public static void main(String[] args) {
  try (ZeebeClient client = ZeebeClientFactory.getZeebeClient()) {

    final ProcessInstanceSearchQueryResponse response1 = client.newProcessInstanceQuery()
        .filter((f) -> f
            .processInstanceKeys(123456L)
            .variable((v) -> v.name("foo").eq("abc"))
            .variable((v) -> v.name("bar").eq("def")))
        .sort((s) -> s.startDate().endDate().asc())
        .send()
        .join();
  }
}

Using the fluent builder, a user can specify a search request fluently. This is useful for simple search requests; it's compact and does not require much code.

Static Builders

public static void main(String[] args) {
  try (ZeebeClient client = ZeebeClientFactory.getZeebeClient()) {

    final ProcessInstanceFilter filter = SearchBuilders.processInstanceFilter((p) -> p
        .processInstanceKeys(123456L)
        .variable((v) -> v.name("foo").eq("bar"))
        .variable((v) -> v.name("bar").eq("def")));

    final ProcessInstanceSort sort = SearchBuilders.processInstanceSort()
      .startDate().asc();

    final var size = 100;

    final ProcessInstanceSearchQueryResponse response2 = client.newProcessInstanceQuery()
      .filter(filter)
      .sort(sort)
      .page((p) -> p.size(size))
      .send()
      .join();

    final ProcessInstanceSearchQueryResponse response3 = client.newProcessInstanceQuery()
        .filter(filter)
        .sort(sort)
        .page((p) -> p.size(size).searchAfter(response2.getSortValues().toArray())) // use sort values from previous response3
        .send()
        .join();

    final ProcessInstanceSearchQueryResponse response4 = client.newProcessInstanceQuery()
        .filter(filter)
        .sort(sort)
        .page((p) -> p.size(size).searchAfter(response3.getSortValues().toArray()))  // use sort values from previous response4
        .send()
        .join();
  }
}

No filter

public static void main(String[] args) {
  try (ZeebeClient client = ZeebeClientFactory.getZeebeClient()) {

    final ProcessInstanceSearchQueryResponse response5 = client.newProcessInstanceQuery()
        .send()
        .join();
  }
}

When a user has a more complex query and/or wants to collect entities in pages, the static builders can be used. They allow the creation of a filter and sorting fluently, which can then be reused for the next requests to get the pages one after the other.

In both cases, the users are not restricted in defining the search request (first sorting, then filtering, and so on). That differs from the existing command API in the Client. There, the commands define one to many steps before executing a command. That way, the user is guided to create a command. IMO, this does not fully apply to search request API, assuming the structure a search request will always have - as mentioned above - it is clear what a user can configure but doesn't have to.

[tmetzke]

I haven't looked at the code yet, will do that tomorrow.

On a logical level, great proposal, @romansmirnov 👍
The fluent builders are quite handy 👍
The static builders should come in handy for discovery of options, i.e. what kind of filters and sorts can I create without having to use constructors of classes I first have to find 👍

Regarding command creation and execution, I think it's fine to have only one level of steps, as they are usually called. The options are all equally optional and can be left out completely. That's like a topology command, with no required options to use, so it's directly a final step. That should be okay.

The only thing that looks a bit off is having to use toArray on the sort values of the previous response. Would be nice if the returned sort values can be consumed by the query API directly. Not sure if there's a specific reason to have different data types here?

[ralfpuchert]

LGTM, found a minor typo. I saw the fluent and builder approach already in demo application code.

[ralfpuchert]

I guess .gt(f.getGt()) was meant here.

[nathansandi]

I liked the approach over the API, looks pretty fluent in terms of proposal to me. 💯 It effectively covers use cases from simple to complex search requests, demonstrating the versatility of the API.

I would agree with @tmetzke in the terms of use toArray on the sort values of the previous response. But it can be improved after it works.

Great work here 👍

[romansmirnov]

Updated the API:

1. Removed the usage of #toArray() to pass searchAfter/searchBefore values
2. Returns a proper response object

final SearchQueryResponse<ProcessInstance> response = client.newProcessInstanceQuery()
        .filter((f) -> f.variable((v) -> v.name("foo").eq(5)))
        .sort((s) -> s.endDate().asc())
        .page((p) -> p.size(20))
        .send()
        .join();

final SearchQueryResponse<ProcessInstance> response2 = client.newProcessInstanceQuery()
        .filter((f) -> f.variable((v) -> v.name("foo").eq(5)))
        .sort((s) -> s.endDate().asc())
        .page((p) -> p.size(20).searchAfter(response.page().lastSortValues()))
        .send()
        .join();

[romansmirnov]

      final SearchQueryResponse<ProcessInstance> response = client.newProcessInstanceQuery()
          .sort((s) -> s.endDate().asc())
          .page((p) -> p.size(20))
          .send()
          .join();

      response.items().forEach(System.out::println);

returns

class ProcessInstance {
    tenantId: <default>
    key: 2251799813685252
    processVersion: 1
    bpmnProcessId: foo
    parentKey: null
    parentFlowNodeInstanceKey: null
    startDate: 2024-06-19T14:35:51.393+0000
    endDate: null
}
class ProcessInstance {
    tenantId: <default>
    key: 2251799813685260
    processVersion: 1
    bpmnProcessId: foo
    parentKey: null
    parentFlowNodeInstanceKey: null
    startDate: 2024-06-19T14:35:51.543+0000
    endDate: null
}
class ProcessInstance {
    tenantId: <default>
    key: 2251799813685268
    processVersion: 1
    bpmnProcessId: foo
    parentKey: null
    parentFlowNodeInstanceKey: null
    startDate: 2024-06-19T14:35:51.591+0000
    endDate: null
}
...

[nathansandi]

[❓] If I am not wrong, on Camunda Services, we refer those fields to sort, why we need re-do it here on the SortImpl?

[romansmirnov]

Camunda Services is a server-side implementation, and the sorting there defines on which (index) field sorting must happen.

This code here is used on the client side, which eventually calls the /process-instances/search endpoint. At this point, the user constructs the search query being serialized into a JSON, for example:

{
  "filter": {
    "processInstanceKey": 22456786958
  },
  "sort": [
    { "startDate": "ASC" }
  ],
  "page": {
    "limit": 10
  }
}

When the Process Instance REST controller receives that request, it must somehow map the client's search request to the actual service, i.e., based on the passed sorting field it will call the respective sort method in the service.

[tmetzke]

Looks really good overall 👍
I added a couple of understanding questions, nothing that really blocks the PR though.

[tmetzke]

❓ We don't have from defined in the REST API guidelines so far. What is this used for?

[romansmirnov]

It can be used for a classic pagination without search after, however, I can remove it if not needed.

[tmetzke]

Understood. Do we offer this on the controller/server side?

[romansmirnov]

Yes, it offers it already.

[tmetzke]

Ah OK, I didn't know 👍 Let's keep it then and add it to the REST guidelines 🙂

[tmetzke]

I didn't add it to the guidelines yet because we weren't sure how complex it would be to add it to the search controllers.

[tmetzke]

@romansmirnov, would you mind adding this to the REST guidelines by creating a proposal, describing how to define and use this in search and response? 🙂 I can take care of documenting this in the guidelines then 👍