Consumer's "RdKafka::ERR__PARTITION_EOF "error not normal

[Z-J-X]

Consumer's "RdKafka::ERR__PARTITION_EOF "error not normal
In the process of consumption, the data is not consumed to the end, but the EOF error is reported.

Test scenarios：

Consumer:

while (run) {
		RdKafka::Message* msg = consumer->consume(1000);

		if (msg->err() == RdKafka::ERR_NO_ERROR)
		{
			std::cout << msg->partition() << ":" << msg->offset() << ":" << static_cast<const char*>(msg->payload()) << std::endl;
		}
		else if (msg->err() == RdKafka::ERR__PARTITION_EOF)
		{
			std::cout << "eof" << std::endl;
			Sleep(10000);
		}
		delete msg;
	}

producer:

bin/kafka-console-producer.sh

Start execution:

Start the consumer program and print the following information first:

% Created consumer rdkafka#consumer-1
RebalanceCb: Local: Assign partitions: testuu[0]:-1001,
partitions size:1
eof

Consumers start to sleep for 10 seconds,Within 10 seconds, the producer starts to produce data,Switching from letter A to s or s to D will have a small time interval,Producer data are as follows

>a
>a
>a
>a
>a
>a 
>s
>s
>s
>s
>s
>s
>d
>d
>d
>d
>d
>d
>
>d
>ff
>f
>f
>f
>f
>
>f
>g
>g
>g
>g
>g
>g

After 10 seconds, consumers began to consume data,The phenomenon is as follows:

% Created consumer rdkafka#consumer-1
RebalanceCb: Local: Assign partitions: xin[0]:-1001,
partitions size:1
eof
0:58936:a
0:58937:a
0:58938:a
0:58939:a
0:58940:a
0:58941:a
0:58942:s
eof
0:58943:s
0:58944:s
0:58945:s
0:58946:s
0:58947:s
0:58948:d
eof
0:58949:d
0:58950:d
0:58951:d
0:58952:d
0:58953:d
0:58954:
0:58955:d
eof
0:58956:ff
0:58957:f
0:58958:f
0:58959:f
0:58960:f
0:58961:
0:58962:f
eof
0:58963:g
0:58964:g
0:58965:g
0:58966:g
0:58967:g
eof

In short, there is still data in the partition, but the consumer reported the EOF error

summary:

I don't think there will be multiple EOF errors when the data produced within 10 seconds is consumed by consumers

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): Releases v1.7.0`
• Apache Kafka version: v2.3.1`
• Operating system: windows10`

[edenhill]

You will receive an EOF each time the consumer reaches the end of the partition when there are no new messages; i.e., when it has fully caught up.
If producers are producing to the topic at the same, and the produce rate is lower than the consume rate, then it is likely that EOF is emitted continously.

[Z-J-X]

I know what you said, but another situation is like this：
When consumers encounter EOF and sleep for 10 seconds, when producers and consumers start at the same time, consumers begin to sleep for 10 seconds when they encounter the first EOF. During this period, producers intermittently produce data and produce 20 pieces of data. After 10 seconds, producers stop producing and consumers start consuming. consumers consume these data, multiple EOFs will be generated, unless consumers close and re consume.

[edenhill]

That's your application sleeping, not the consumer, though right?
If you want the consumer to "sleep", use pause().

[Z-J-X]

Thank you very much☺. This way can solve this problem.I want to use sleep to simulate the logic of data processing after consuming data,If the data processing time is too long, it will cause the problem I mentioned earlier: consumers report EOF error before consuming to the end of partition.Why is this? Is it because consumers' EOF information is not synchronized with Kafka's EOF information? If you consume a piece of data, then execute pause (), then process the data, and then execute resume (), then everything will be normal