When a kafka node goes down in single-replica topic mode, the Kafka indexing task cannot be started

[wanglihui-git]

Affected Version

0.18.1——0.22.1

Description

Due to the large amount of data in the production environment, our kafka cluster had to use a single-replica topic. When a kafka node goes down, the kafka indexing task cannot be started. The normal running Supervisor can still run continuously, but after the reset operation, it can't run either.

If this happens in the production environment, and the kafka node is down and cannot be recovered in a short time, how can the Druid task increase the reliability of it?

The following is a screenshot of my test. The error message is: 'Timeout of 60000ms expired before the position for partition topic-0 could be determined'.After a while, the Supervisors state changed to 'LOST_CONTACT_WITH_STREAM'.

Thank you!

[abhishekagarwal87]

There is nothing you can do on the druid side to increase reliability if the data source itself is unavailable. aren't the producers running into the very same problem that druid is running into?

[wanglihui-git]

There is nothing you can do on the druid side to increase reliability if the data source itself is unavailable. aren't the producers running into the very same problem that druid is running into?

Kafka node downtime seems to have no effect on the producer, we are using kafka 1.0.0.

[abhishekagarwal87]

Interesting. are you using custom timeout and ack settings on producer?

[wanglihui-git]

Interesting. are you using custom timeout and ack settings on producer?

Yes, we set timeout, and ACK = 1.
For this issue, we can't do too many things, is it because Druid uses a low-level API to connect Kafka? This partition cannot be connected when a partition of Kafka is damaged.
When using Kafka High-Level API, even if there is a partition damage, we can also consume through existing group.id, just unable to add new group.id.

[FrankChen021]

@wanglihui-git This confused me. How could a producer send messages to this single-replica topic if the Kafka node, more precisely the leader of this replica, go down?

[wanglihui-git]

@FrankChen021 The producer will rebalance the message to the normal partition, of course, some data may be lost in the process. But the overall data flow will not be affected, just have a big impact on consumers.

[FrankChen021]

Oh, you're right. I took the single replica as single partition.

The reset operation is supposed to delete the partition information from the metadata storage, and then restart the kafka consumer. If a normal Kafka consumer can consume the messages from the rest partitions, I think Druid's consumer can consume too. Because Druid also uses the high level APIs on KafkaConsumer to pull messages. The only difference between a Druid's consumer and tranditional use of KafkaConsumer is that Druid does not call commit method to commit the consumed offset back to Kafka cluster but only store it in Druid's own metadata store.

Since this problem occurs on your environment, could you check:

1. if the metadata is cleared after the 'reset' operation
2. the logs of overlord after the 'reset' opreation to see any warning or errors.

[abhishekagarwal87]

AFAIK, the producer doesn't rebalance the message to another partitionitself. Say you are partitioning on a field called userId and you have 60 partitions. If the producer determines that a user XYZ is supposed to land on partition 5, then the producer can't push it to partition 6. That can break many semantics within the application.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.