A lot of memory being held by org.bouncycastle.crypto.util.dispose.DisposalDaemon

[vhbcm]

We are seeing a lot of memory being held by Bouncy Castle FIPS 2.1.0. See the attached screenshot from heap dump analysis:

We did not observe the same behaviour with 1.0.x BC FIPS.

It seems all of the objects held are like this:

[mwcw]

I need to provide some background first.

The DisposalDaemon listens for notifications that an object is available to be garbage collected and releases any native allocations associated with that object.

The original behaviour was to release native allocations as soon as a notification was received, however on busy JVMs especially where the machine has many (16+ cores) the JVM will flag an object as being available for collection while other threads are finishing up their last method call to that object.

If the last method call involved calling a function that operated on an native allocation then the JVM would SegFault because because the pointers would be invalid.

In the end the only way we could work around this was to delay cleanup by a fixed period of time.

This delay is adjustable down to zero but the smaller you make the delay the bigger the risk you may run into a SegFault, I can't advise you what the best delay is because it is going to depend on usage and the hardware it is running on.

You can use the property, org.bouncycastle.native.cleanup_delay to set the delay in seconds, the default is five.

Thanks

MW

[wilx]

It seems to me this is a quality of implementation issue. This should be solvable, on Java 9+ at least, without resorting to delaying tactics, using java.lang.ref.Reference.reachabilityFence() around the native calls.

[dghgit]

2.1.0 has to support Java 8. It's possible, but not clear, that going multi-release could make a different approach possible, but at the time developing 2 different blocks of code for handling the gc issue in a multi-release fashion was well beyond the resources we had available to us.

[wilx]

Doug Lea had an implementation for the reachability fence. Even using synchronized would be cheaper than using the whole delaying scheduler ensemble. Implementing this as two separate classes, one default for Java 8 and another as for version 9 in Multi-Release JAR would improve things a lot.

[tmueller64]

Is there any update on a schedule for getting this fixed? We are seeing the retained memory for the DisposalDaemon object growing to 285 MB after about a week of running bc-fips within Tomcat. Is there any workaround that can be used to release the memory? Thanks.

[dghgit]

Try what's in: https://downloads.bouncycastle.org/betas/bc-fips-2.1.1-SNAPSHOT.zip

We think we may have caused some GC confusion by using a private inner class. This should fix it (assuming that is the problem).

[tmueller64]

I ran our application on Tomcat 10 using the 2.1.1-SNAPSHOT build above and the JVM memory problem is solved. Thanks.

[dghgit]

That's a relief! 2.1.1 has been published to Maven Central, note the release is not officially certified, and should be treated as an "update stream" release for FedRAMP purposes. We're talking to our lab about making sure it's officially recognized.