Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Able to add metadata to the committed offset #563
I would like to add metadata based on the ConsumerRecord along with each commit. The issue for this is here: #561
This PR will allow users to pass in a function to the source constructor which will add metadata to each commit based on the consumer record. Each committed offset is guaranteed to contain metadata constructed from the message as long as the commitRefreshInterval is infinite.
If the commitRefreshInterval is not infinite, then when the Consumer is assigned a partition it will also receive an offset without a message. If it has not processed a new message before the end of the interval, it will recommit this offset for this partition without adding metadata.
The code looks good now, but we'd like some documentation, as well.