Requests to activate jobs may result in infinite execution from the Gateway to the brokers

[romansmirnov]

Describe the bug

Whenever the failed attempts (i.e, InFlightLongPollingActivateJobsRequestsState#failedAttempts) are reset, all currently active requests are added to the queue of activeRequestsToBeRepeated:

https://github.com/camunda-cloud/zeebe/blob/12eeb3f81af0c1b65d2257cab52b71a712562757/gateway/src/main/java/io/camunda/zeebe/gateway/impl/job/InFlightLongPollingActivateJobsRequestsState.java#L40-L45

The failed attempts are reset when (1) jobs are activated successfully, and (2) whenever the broker notifies the gateway about newly available jobs.

In the case, the brokers respond with no activated jobs, the Gateway checks the activeRequestsToBeRepeated queue if the request should be repeated immediately or not. That means

1. as long as an active request x is present in the queue activeRequestsToBeRepeated,
2. no jobs are activated with request x, and
3. the failed attempts are lower than the configured failedAttemptThreshold

then the request x will be executed over and over again (see line 163):

https://github.com/camunda-cloud/zeebe/blob/12eeb3f81af0c1b65d2257cab52b71a712562757/gateway/src/main/java/io/camunda/zeebe/gateway/impl/job/LongPollingActivateJobsHandler.java#L157-L167

Scenario 1: Duplicates in activeRequestsToBeRepeated when other requests succeeds

Given three activate requests x, y, and z:

1. send request z to broker -> current activeRequests = [ z ] and activeRequestsToBeRepeated = [ ]
2. send request y to broker -> current activeRequests = [ z, y ] and activeRequestsToBeRepeated = [ ]
3. send request x to broker -> current activeRequests = [ z, y, x ] and activeRequestsToBeRepeated = [ ]
4. request z completes with at least one activated job -> reset failed attempts -> current activeRequests = [ y, x ] and activeRequestsToBeRepeated = [ y, x ]
5. request y completes with at least one activated job -> reset failed attempts -> current activeRequests = [ x ] and activeRequestsToBeRepeated = [ x, x ]

Scenario 2: Duplicates in activeRequestsToBeRepeated on multiple notifications

Given is a request x:

1. send request x to broker -> current activeRequests = [ x ] and activeRequestsToBeRepeated = [ ]
2. Broker notifies the gateway about new available jobs -> reset failed attempts -> current activeRequests = [ x ] and activeRequestsToBeRepeated = [ x ]
3. Again, Broker notifies the gateway about new available jobs -> reset failed attempts -> current activeRequests = [ x ] and activeRequestsToBeRepeated = [ x, x ]

Scenario 3: Another request wins

Given are requests [z1, z2, z3, ...., zn] and x:

1. send requests [z1, z2, z3, ...., zn] and x -> current activeRequests = [ z1, z2, z3, ...., zn, x ] and activeRequestsToBeRepeated = [ ]`
2. request z1 completes with at least one activated job -> reset failed attempts -> current activeRequests = [ z2, z3, ...., zn, x ] and activeRequestsToBeRepeated = [ z2, z3, ...., zn, x ]
3. request x completes without any jobs -> retry request -> current activeRequests = [ z2, z3, ...., zn, x ] and activeRequestsToBeRepeated = [ z2, z3, ...., zn ]
4. request z2 completes with at least one activated job -> reset failed attempts -> current activeRequests = [ z3, ...., zn, x ] and activeRequestsToBeRepeated = [ z3, ...., zn, x ]
5. request x completes without any jobs -> retry request -> current activeRequests = [ z3, ...., zn, x ] and activeRequestsToBeRepeated = [ z2, z3, ...., zn ]
6. ...

In all scenarios, the queue activeRequestsToBeRepeated contains the request x at least once before the response arrives. In a nutshell, as long as the failed attempts are lower than the configured threshold and the request x is present in the queue of repeatable requests, the request x is executed in a loop when it responds with any activated job.

Additionally, the property requestTimeout coming along with the activate request might not be considered in the scenarios above. Meaning, the provided requestTimeout is only considered when

1. the request responds without any activated job, and
2. the failed attempts are greater than the configured failed attempts threshold.

As long as none of these conditions are fulfilled, the request will be in an execution loop and the client won't receive any response until the client closes the connection/request after a certain timeout.

To my understanding, the current implementation does not behave as documented in the API:

https://github.com/camunda-cloud/zeebe/blob/12eeb3f81af0c1b65d2257cab52b71a712562757/gateway-protocol/src/main/proto/gateway.proto#L25-L28

To Reproduce

Expected behavior

• The implementation behaves as documented in the API, see Gateway does not respect the long polling timeout received along with the activate jobs request #8389
• At a minimum, the queue activeRequestsToBeRepeated should not contain duplicates, see InFlightLongPollingActivateJobsRequestsState#activeRequestsToBeRepeated contains duplicates #8390

Environment:

• Zeebe Version: 1.2.5

---

is related to #8267
is related to #7659
depends on #8389
depends on #8390

[romansmirnov]

@deepthidevaki and @npepinpe,

Based on the observations made in #8267, I spent some time to get a better understanding of the long-polling implementation. To my understanding, it does not behave as documented in the "API" which might cause different issues like the request might be executed in a "loop"

https://github.com/camunda-cloud/zeebe/blob/12eeb3f81af0c1b65d2257cab52b71a712562757/gateway-protocol/src/main/proto/gateway.proto#L25-L28

Do I miss something? And if not, should we adjust the implementation according to the documentation or the other way around? Should we consider this for this quarter Q4?

[Zelldon]

I think this is related or superset of #7659

[romansmirnov]

@Zelldon, thanks for linking issue #7659. It's actually quite an interesting scenario, you raised in the issue. However, but I don't think they are related, i.e., in my opinion, fixing this issue will not solve the problem described in #7659.

[deepthidevaki]

Is this already fixed? Two related issues are already closed. Is there anything else to be done here?