Question on memory growth after consumer unsubscribes topics

[esselgee]

I have an application which uses the KafkaConsumer C++ class from librdkafka 1.8.2. My consumer code subscribes and unsubscribes to various topics based on messages received in one of the topics. I have a Rebalance callback object that does the assign and unassign operations as in the examples.

I have a test that will continuously subscribe and then unsubscribe to a unique topic name each time. There are no messages in these topics and they are deleted from the broker after the unsubscribe (by another process).

I'm seeing that the memory usage of my application continuously increases during this test. Looking into it further, I enabled the librdkafka logging. I see many references to topic names long after their subscription has ended. It seems that the metadata code is keeping track of topics even after their subscriptions have been removed possibly. From the statistics event, I see the field "metadata_cache_cnt" is around 700 when I stopped the test. This seems to correlate to the number of topics that had been subscribed for, but now are not subscribed for.

So is this the expected behavior? I don't understand why the library is maintaining a list of all of these topics that are no longer used. Is there some property that needs to be set or something I need to call in order to clean all these topics up and reduce memory consumption?

Thank you.