New minimumIdle connections every maxLifetime causing MySQL'AbandonedConnectionCleanupThread Memory Leak

[zeeshen]

Environment

HikariCP version: 3.4.1
JDK version     : 1.8.0_111
Database        : MySQL
Driver version  : mysql:mysql-connector-java:jar:8.0.17

---

We've noticed a memory leak in a spring-boot application:

With the help of heap dump and MAT, we found that the memory leak's caused by

Looking into the code base, we found that HikariPool closes the old connections and creates new ones every maxLifetime. But these closed connections are not garbage collected correctly by the AbandonedConnectionCleanupThread of MySQL .

We can produce the phenomenon by setting the maxLifetime to a relatively small value like a minute(all other configs are using the default value). Using tools like VisualVM, we can see an obvious increasement of AbandonedConnectionCleanupThread$ConnectionFinalizerPhantomReference objects.

We're not sure that it's a Hikari issue or a mysql-connector issue actually. As submitting an issue to MySQL project is a little complex, we really hope that you can take a look at this issue. Thank you so much.

[quaff]

I think it's nothing about Hikari, you can replace mysql-connector-java with latest mariadb-java-client and compare.

[quaff]

Number of ConnectionFinalizerPhantomReference will shrink to minimumIdle after GC.

[zeeshen]

Number of ConnectionFinalizerPhantomReference will shrink to minimumIdle after GC.

Actually it's not I think, or the heap won't keep growing all the time.

[adamsau]

similar problem exists with mysql:mysql-connector-java:jar:8.0.16 and spring boot and openjdk 1.8.
memory keeps going up, ConnectionFinalizerPhantomReference set size keeps increasing without shrinking. jvm force GC does not help.

[brettwooldridge]

@rivenhk @zeeshen It could very well be a bug in the driver, as there are several examples of environments (mainly application containers) where the MySQL driver misbehaves:

https://issues.apache.org/jira/browse/NIFI-5501

https://bugs.mysql.com/bug.php?id=69526

One possible answer/lead is here:

https://stackoverflow.com/questions/25699985/the-web-application-appears-to-have-started-a-thread-named-abandoned-connect

[bharath2109cred]

what solved this issue @zeeshen ? facing similar issue

[brettwooldridge]

@bharath2109cred What version of mysql-connector-java are you using? You may want to investigate the relatively new property (mysql) Java system property com.mysql.cj.disableAbandonedConnectionCleanup.

https://bugs.mysql.com/bug.php?id=96870

[sinsuren]

Thanks @brettwooldridge we were also struggling with same issue in our application.

[brettwooldridge]

Make sure you are on the latest version of Connector/J, and set system property

-Dcom.mysql.cj.disableAbandonedConnectionCleanup=true.

[sinsuren]

Posting this for future users.

We had one possible memory leak into our application due to which old generation in our application was never coming down.

Configuration of application.
8 Core + 6 GB heap size + java8 + G1GC + Caffeine for local cache + Hikari to manage db thread pool + mysql-connection 8.0.17

When we dig deeper found one configuration miss into Caffeine due to which expired keys were not getting evicted, After fixing this issue we took heap dump of application and found a quite good number of reference of above object.

We also tried reproducing above issue on local with very less lifespan of db connection. Quite a good number of those references were getting generated and after moving to latest connector and setting -Dcom.mysql.cj.disableAbandonedConnectionCleanup=true in java params, stopped those references.

But if application is using hikari or any other connection pool manager, with under right configuration those references will keep on getting collected in timely manner, and should not be trouble. In our application we are having maxlifetime of
30 minutes. So those references are not creating any issue and getting collected when a major/mix GC is fired.
PS: we have to tune our GC params as well a bit to fire mix gc in timely manner.