java: linear solver memory leak

[mtetreau]

java: heap memory consumption
version: 9.4

There is an issue regarding the implementation of ortools-java classes com.google.ortools.linearsolver.MPConstraint and com.google.ortools.linearsolver.MPVariable.

Both classes override the deprecated method finalize() resulting in a significant and unnecessary consumption of heap memory and process time. A 50% reduction of heap memory and 20% reduction of process time have been observed in our tests when removing those methods, since related instances were not map nor treated by the jvm finalizer.

Since in java, there is no control on when finalize() is called by garbage collection, a better practice to clean resources would probably be to explicitly call MPVariable.delete()/MPConstraint.delete() if needed.

Thank you for ortools!

[lperron]

this is swig generated code. Not sure what we can do.

[lperron]

Can you try with com.google.ortools.modelbuilder ?

The API is a bit different. It has 2 classes (model and solver). Model manipulation is very limited. But it should be lightweight, and all the modeling layer if pure java (outside of the model class).

[mtetreau]

Unfortunately, our solver connector is too generic and cannot be easily simplified to use modelbuilder.

Thank you, it allowed us to learn about this api!

[Opsiana]

Hi Laurent --
I am afraid that I am seeing an issue that is consistent with what mtetreau describes. We are running or-tools (v 9.7.2996) with Java 17 in a Docker container on GCP cloud Run. The memory utilization profile of the container goes up when a large MIP instance is solved, and does not revert to initial levels after the request is completed. I have used a heap profiler to confirm that a lot of memory remains allocated to MPvariable and MPconstraint. Eventually the container crashes due to out-of-memory.

[yasha-spare]

I'm seeing similar behaviour:

• App is a JVM vehicle routing problem solver
• In one stage of the solving, that gets run very regularly, it solves a small linear equation, using com.google.ortools.linearsolver.MPSolver.createSolver("CLP")
  • These are pretty small problems, created and solved in about 1ms each, there's just an absolute tonne of them being solved over time
• If the app gets too busy, all the memory starts to get consumed by OR tools MPVariable and MPConstraint instances:

Heap dump:

java.lang.ref.Finalizer is OR-tools stuff too:

Like @Opsiana I'm also on 9.7.2996 of ortools-java, and Java 17. Seems like a memory leak, but it's very clear from the code I'm not keeping around any references to OR tools objects - the model gets built, solved, then immediately transformed into a totally different representation, with no references to the OR tools classes hanging around.

Anyone found a workaround/solution? I don't think this problem can be represented with modelbuilder, though I'm trying. We have also previously tried to use GLOP without success for our linear program - don't know if this would behave better than CLP or not?

FWIW, if any OR-tools maintainers wanted to hop on a video call at some point, pair debug/profile together, I'd be happy to do so!

[yasha-spare]

For anyone else reading this - we ended up solving this by switching off of or-tools, to clp-java (https://github.com/quantego/clp-java). We were using CLP under the hood anyways with or-tools, so they're both just Java wrappers of the same underlying native lib. Behaviour and performance seems to be identical between the two (for our use case, at least), but with clp-java memory usage stays constant and low, even under really heavy load, while with or-tools under identical load memory climbed straight up until OOM.

[ClaudioConsolmagno]

I have ended up solving my issue 2 ways: I did move to ModelBuilder (is there Java docs on it btw? Had to figure out lots of things myself), and it did help a lot. Instead of my server crashing about twice a day, it started crashing once every 2 days.

So moving to ModelBuilder alone did not fix the issue. What really did fix it (and it's been running fine for over a week with a lot more reasonable memory consumption) was setting these jvm flags: -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000. Have also set env flag MALLOC_ARENA_MAX = 2 but not sure that was strictly necessary.

As I understand, the native code under or-tools uses the malloc OS function to allocate large areas of memory (arenas) and the default malloc doesn't let go of memory very easily with some sort of fragmentation problem. Setting -XX:TrimNativeHeapInterval=60000 runs a "trim" command every minute and that sort of forces unused off-heap memory to be released.

So if you are running the solver lots of times, memory gets allocated and de-allocated within those arenas but don't necessarily get released back to the OS due fragmentation and some technical reasons I don't fully understand. The trim forces the arenas to be locked and release memory back to the OS, so the locking has some downside which has been unoticeable in my use-case.

This flag is only available since Java 17 so if you're on <17, setting MALLOC_ARENA_MAX = 2 (default is generally 4 afaik) may help. But from my research the best thing to do regardless of Java version is to setup a different allocator function like jemalloc (described in a devoxx talk in the youtube link below). That should work even better than the flag and works for any Java version. I didn't go there though since the flag was enough for my use case.

Some resources if anybody wants to know more:

https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory/
https://stackoverflow.com/a/28935176/9994213
openjdk/jdk#14781
openjdk/jdk#17799
prestodb/presto#8993
https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior
redhat-developer/vscode-java#3395
https://www.youtube.com/watch?v=c755fFv1Rnk&t=2046s

I hope this helps and maybe even helps or-tools developers find a way to get around this issue.

[tkz2020]

The CpSolver we use has always had memory leak problems. Has anyone encountered similar problems?

[lperron]

CP-SAT? Le mar. 23 avr. 2024, 04:13, tukangzheng ***@***.***> a écrit :

…

The CpSolver we use has always had memory leak problems. Has anyone encountered similar problems? — Reply to this email directly, view it on GitHub <#3684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUPL3JEVAI4URWDA5PDKS3Y6W7WFAVCNFSM6AAAAAAVDXDZNCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZRGI4DENJYG4> . You are receiving this because you commented.Message ID: ***@***.***>

[tkz2020]

yes

[lperron]

CP-SAT does not leak memory. It uses a lot of memory, but does not leak. Le mer. 24 avr. 2024, 10:40, tukangzheng ***@***.***> a écrit :

…

yes — Reply to this email directly, view it on GitHub <#3684 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUPL3I7ES62TJGG26PQWM3Y65VXBAVCNFSM6AAAAAAVDXDZNCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUGQYDOOJRG4> . You are receiving this because you commented.Message ID: ***@***.***>

[mtetreau]

@Mizux, from what we have observed, it was not a memory leak.

Just an avoidable consumption of memory used by the java.lang.ref.Finalizer thread and triggered by the presence of the finalize() method on MPConstraint/MPVariable classes.

May be the issue title should then be: 'java: linear solver memory bloat'