Store offset commit metadata when calling rd_kafka_offsets_store

[mathispesch]

Fixes #3927.

Offset commit metadata can be used to send additional data to the broker when making an offset commit. See this doc from the Java consumer.

Librdkafka currently sends the offset commit metadata when the commit is requested by calling rd_kafka_commit_queue but not when calling rd_kafka_offsets_store to use the auto-commit. This PR fixes that by storing the offset commit metadata when calling rd_kafka_offsets_store.

[mathispesch]

Hey @edenhill or @ijuma,
Could you please give this PR a look? 🙂

[ijuma]

cc @emasab

[mathispesch]

Hey folks,
Have you had the chance to look into this PR?

[milindl]

Looks good to me, besides the tiny issue with the test. I'd like @emasab to weigh in.
We should add back the CHANGELOG, but under the next version, (we can use 2.1.0, or just call it vNext or something).

[milindl]

This needs to be destroyed at the end of the test, rd_kafka_topic_partition_list_destroy(committed_toppar);

[mathispesch]

Good catch! Fixed this in 5d37ab1.

[emasab]

@mathispesch Thanks for the contribution! For security reason currently it's not allowed to run SemaphoreCI on forked PRs. I merge it into an internal branch and open a new PR. If you want to do additional changes before merging to master please open a new PR based on that branch.

[mensfeld]

@mathispesch @emasab should it also store metadata when using rd_kafka_offsets_store and then rd_kafka_commit with null or not? 🤔