Memory leak in EventDeserializer caused by tableMapEventByTableId

[TengHuo]

Hi @osheroff

There is a long ago memory leak issue in EventDeserializer caused by tableMapEventByTableId. I saw you fixed it by using a bounded map which has size limit of 10000

code here:

mysql-binlog-connector-java/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventDeserializer.java

Line 69 in a93b785

this.tableMapEventByTableId = new LRUCache<>(100, 0.75f, 10000);

It can avoid the huge memory usage when there are large amount of tables in one MySQL instance, but it still could take a lot of memory if this map is almost full. We got this issue in Debezium.

However, according to this post: https://dba.stackexchange.com/questions/51873/replication-binary-log-parsingtableid-generation-on-delete-cascade-handling
(a better investigation post but in Chinese: http://blog.chinaunix.net/uid-26896862-id-3329896.html)

Table id is not a constant value in one MySQL instance, it will assign a new id when there is no cached table id found, or roll to a new id when the cache in MySQL server my_hash_search_using_hash_value is full. Thus, it is not necessary keeping this tableMapEventByTableId map all the time.

I have a fix for this issue by just cleaning this map regularly after receiving a MySQL binlog ROTATE event (definition here: https://dev.mysql.com/doc/internals/en/event-meanings.html)

[TengHuo]

BTW
This is Maxwell code for cleaning table map cache:

https://github.com/zendesk/maxwell/blob/04dfa39657c432a47c654d50b76814d5f8f7ce32/src/main/java/com/zendesk/maxwell/replication/BinlogConnectorReplicator.java#L678

[TengHuo]

Related issue in Maxwell: zendesk/maxwell#1468

[TengHuo]

Sorry for forgot mention the detail about the screenshot, it was found in Debezium, which is a similar leak in MySqlStreamingChangeEventSource.java

code here: https://github.com/debezium/debezium/blob/48b9d7d7c29065ce20ce3f3f8de363d38011fb6f/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java#L224

In Debezium, there is another tableMapEventByTableId which is used in eventDeserializer, and this map has never been cleaned in MySqlStreamingChangeEventSource.java. So it causes the memory leak like the original EventDeserializer

[osheroff]

this doesn't seem super important to me. the LRU cache doesn't grow over 10,000 entries... so while it's a "leak" in a sense, it's an intentional one and shouldn't really cause much real-world memory usage. If you look at the original maxwell issue, you'll see that each entry in this map averaged 50 bytes -- so 10k entries, 500k memory usage, I'm not super worried about it.

Your patch seems kinda/sorta fine too, the only thing I'd be worried about is if there was an object that hadn't been deserialized yet holding on to the table map hash at the time we called .clear(). I don't necessarily think that that can happen in the real world, but i'd need to read the code deeper.

[TengHuo]

Hi Osheroff, thanks so much for your reply.

I just dig more detail in Debezium and Maxwell source code, I totally agree with you. It's shouldn't be a problem in Maxwell since there is an upper limit in LRU cache, and Maxwell did clean their TableCache regularly.

In Debezium, it is totally different. It inherit EventDeserializer and put another tableMapEventByTableId in nextEvent method, but there is no cleaning for this map. Then, it causes memory leak as I mentioned above. But we can't do anything about it in mysql-binlog-connector-java code for this. It should be fixed in Debezium code.

And about my approach for fixing this issue, it did never happen that the map is doing clear() while a new event arrive. We have been using this fix for a year in an old version of mysql-binglog-connector-java from https://github.com/shyiko/mysql-binlog-connector-java. And you are free to close this issue and PR if you think it is not necessary.

BTW, do you know how to raise an issue in Debezium community? looks like they disabled Github issue function and using Redhat issue tracking, but I can't find how to fire a new ticket in it😂

[osheroff]

ah, so yeah. we could fix it here, but debezium would still have to update their dependencies.

I can't see how to open a debezium ticket either. ahhh jira. they're co-maintainers of this repo, maybe they'd respond to a mention here? I don't want to mess with how they engage with their user base, but...

[TengHuo]

Got it, np. let me feedback the tableMapEventByTableId issue of Debezium in their community. thanks a lot for your help