GH-2806 : Receiving an empty list when using RecordFilterStrategy on batch messages

[chickenchickenlove]

Motivation:

• Receiving an empty list when using RecordFilterStrategy on batch messages #2806
• In the current batch mode, even if the RecordFilterStrategy filters all records resulting in an Empty List being returned, the KafkaListener is still invoked. In contrast, in single record mode, if record are filtered, the KafkaListener is not called. This difference in behavior between the two modes can cause confusion for users.

Modifications:

• Add public method isAnyManualAck() to Acknowledgment to verify that manualAck is needed on FilteringBatchMessageListenerAdapter.
• Modify FilteringBatchMessageListenerAdapter.
  • add field consumerAware as final (IMHO, we don't need to calculate it every single call onMessage().
  • add logic (if empty list and manual Ack == true, KafkaListener will be invoked. if empty list and manual Ack == false, KafkaListener will not be invoked even if listener is kind of ConsumerAware. In detail, See Discussion section below.)

Result:

• Closes Receiving an empty list when using RecordFilterStrategy on batch messages #2806
• When the RecordFilterStrategy filters all records and returns an Empty List, the KafkaListener is invoked only if it is in manual ACK mode.

Discussion

• When using a ConsumerAware Listener, commits can be made using Consumer.commitSync() and Consumer.commitAsync(). However, when using a ConsumerAwareAckListener, it seems possible that commits using the Consumer and commits using Ack could be processed simultaneously. That situation seems quite ambiguous.

[chickenchickenlove]

I see! i reverted all and make new commits.

[artembilan]

I think the problem comes from the MethodKafkaListenerEndpoint.createMessageListenerInstance():

		if (isBatchListener()) {
			BatchMessagingMessageListenerAdapter<K, V> messageListener = new BatchMessagingMessageListenerAdapter<>(
					this.bean, this.method, this.errorHandler);

where we have:

public class BatchMessagingMessageListenerAdapter<K, V> extends MessagingMessageListenerAdapter<K, V>
		implements BatchAcknowledgingConsumerAwareMessageListener<K, V> {

and that leads to the:

		if (listener instanceof AcknowledgingConsumerAwareMessageListener
				|| listener instanceof BatchAcknowledgingConsumerAwareMessageListener) {
			listenerType = ListenerType.ACKNOWLEDGING_CONSUMER_AWARE;
		}

So, the logic in that FilteringBatchMessageListenerAdapter always falls to the consumerAware as true.
Therefore sounds like we cannot achieve the requested logic with existing flags.
Not sure, though, if that would be convenient to introduce a new one exactly for this use-case for batch filtering.

Maybe RecordFilterStrategy could be improved with extra boolean method to implement?
Like:

default boolean ignoreEmptyBatch() {
    return false;
}

[chickenchickenlove]

Thank you for your analysis a lot 🙇‍♂️🙇‍♂️🙇‍♂️🙇‍♂️!
The logic remains the same eventually, but I think the direction you proposed is better because it gives users a choice.

[chickenchickenlove]

I make new commit to apply your reviews.

	default boolean ignoreEmptyBatch() {
		return false;
	}

I added ignoreEmptyBatch() to RecordFilterStrategy interface, and use it on FilteringBatchMessageListenerAdapter instead of consumerRecords.isEmpty().

This way, those who want to be as it is can do so without modifying their codes. Meanwhile, it provides a choice for users who have considered this to be an issue until now.

What do you think?
When you have free time, take a look please 🙇‍♂️

[artembilan]

We need some tests for this new feature. And, yes, I also think that this should go to 3.3 already . So, please, fix Javadoc respectively. And add some doc, too.

[chickenchickenlove]

@artembilan , thanks for your comments 🙇‍♂️

We need some tests for this new feature. And, yes, I also think that this should go to 3.3 already . So, please, fix Javadoc respectively. And add some doc, too.

I added a couple of test cases to test new public API.
When you have free time, please take a look 🙇‍♂️

[chickenchickenlove]

I added spring-kafka-docs as well to Filtering Messages section.

[artembilan]

The review is a bit thorough, but there is no rush with the fixes.
We have like a month yet until with start a new 3.3 version.

Thanks