LogStorageAppender Actor occupies an Actor Thread forever due to a full backpressue queue

[romansmirnov]

Describe the bug

The LogStorageAppender subscribes to the write buffer (i.e., Dispatcher) to read from it and to append the available fragments to the LogStorage:

https://github.com/camunda-cloud/zeebe/blob/73e5c7be9f453e30b5aebc07aae322ee5f82b11e/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/LogStorageAppender.java#L151-L154

However, before appending to the LogStorage, the LogStorageAppender tries to acquire a "token" from a limiter (i.e., backpressure queue), and only if a token got acquired it appends to the LogStorage (and the read fragments are marked as read in the write buffer). On the other side, if no token got acquired, the fragments are not appended to the LogStorage (and the fragments are still marked as not read in the write buffer):

https://github.com/camunda-cloud/zeebe/blob/73e5c7be9f453e30b5aebc07aae322ee5f82b11e/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/LogStorageAppender.java#L119-L136

So whenever some fragments are available on the write buffer, the Actor LogStorageAppender is submitted to the broker thread group's task queue so that a broker thread can execute this Actor eventually. When executing this Actor, it checks whether there are some fragments are available on the write buffer:

https://github.com/camunda-cloud/zeebe/blob/73e5c7be9f453e30b5aebc07aae322ee5f82b11e/util/src/main/java/io/camunda/zeebe/util/sched/channel/ChannelConsumerCondition.java#L43-L48

and if there are some fragments available, it will execute the corresponding Actor Job (i.e., read from the write buffer and try to append to the LogStorage).

The actor job is executed as long as ChannelConsumerCondition#poll() returns true.

In a scenario, where the appender's backpressure queue is full (i.e., no token can be acquired from the limiter), the Actor Thread keeps executing the actor job because ChannelConsumerCondition#poll() still returns true. And as long as the backpressure is not emptied the Actor Thread will continue executing that job forever.

For example, this can happen in the following case:

1. For a certain partition - where the broker is the leader, the appender's backpressure queue is full (so that no token can be acquired from the limiter)
2. The Actor Thread executes the LogStorageAppenders actor job, i.e., it reads from the write buffer and tries to append the LogStorage (and ChannelConsumerCondition#poll() returns always true, but no token can be acquired)
3. For the same partition, the Raft layer transitions from leader to follower. This submits an Actor Task to ensure that the Zeebe application layer transitions to follower as well (i.e., to ensure that corresponding services are stopped and started).
4. The submitted Actor Task gets assigned to the same thread task queue as the LogStorageAppender is already in.

As a consequence, the submitted Actor Task to transition to follower on the Zeebe application layer won't be executed, because the Actor Thread is already occupied by the LogStorageAppender Actor which never releases the Actor Thread.

Note: This may also happen in other scenarios in which no leader change happens.

What is the impact of that issue?

In the worst case, it can happen that all Actor Threads are occupied by such Actor's (but for different partitions). This results in:

• The broker being stuck in those Actor Jobs and never being able to make progress with any other Actor
• Growing Actor Job Queues, as long as others (e.g., Raft) are able to submit jobs to the Actor Job Queues. That may result in OOMs.
• ...

Expected behavior
The LogStorageAppender Actor releases the Actor Thread so that other Actor Jobs can be executed.

Possible Solutions

• When no token can be acquired because the backpressure queue is full, then resubmit the Actor Task to the actor thread task queue.
• When executing ChannelConsumerCondition#poll() take the backpressure queue into account.

Environment:

• Zeebe Version: 1.3.0

---

related to https://jira.camunda.com/browse/SUPPORT-11966

[Zelldon]

I played today a bit around with metrics and actor and maybe if you know how to reproduce it this could help to see this. Currently it doesn't look like this happens in normal runs.

This graph shows how often a certain actor is called in a second.