[Feature Request]: Improved Mechanism for Signalling Message Processing Completion

[filipeesch]

Is your request related to a problem you have?

There is currently no standardized and flexible mechanism in the framework to signal when message processing ends for different types of consumers. This inconsistency can lead to message loss, particularly in cases where an application is abruptly terminated or killed. This issue only happens when using custom async processing middlewares or libraries like TPL Dataflow, as it becomes challenging to manage message lifetimes effectively.

Describe the solution you'd like

Leveraging the IConsumerContext interface, I propose the following enhancements to streamline the mechanism for message completion:

1. Method and Property Renaming:

• Rename the StoreOffset() method to Complete(), which provides more clarity about its function.

2. Handling Message Completion:

The Complete() method should be invoked to signal the end of message processing:

void Complete();

3. New Property: Completion

Introduce a new property named Completion:

Task<TopicPartitionOffset> Completion { get; }

This Completion task will be completed when the Complete() method is called, signaling that message processing has ended. It will allow asynchronous operations to await the end of message processing.

4. Introduce the AutoMessageCompletion Flag

Create a property named AutoMessageCompletion:

bool AutoMessageCompletion { get; set; }

This flag determines whether the framework will automatically signal the completion of a message once it's processed.

This will ensure that message processing is marked as finished and that associated resources are correctly released.

Summary:

With these changes, the framework can offer a more versatile mechanism for managing message lifetimes, especially beneficial for consumers employing unique approaches like TPL Dataflow or the BatchConsume middleware.

Are you able to help bring it to life and contribute with a Pull Request?

Yes

Additional context

This enhancement will not only benefit consumers with unique message processing requirements but also simplify core framework code by providing a standardized way to signal the completion of message processing.