SCIP gets stuck after upgrade to 9.2.4

[lhoegh]

Background

• We build models programatically and solve them on a job server, that uses scip through the jscip interface.
• Our models are MIP's with lots of binary variables. They generally take somewhere between 0 to 5 minutes to solve, and have a time limit of no more than 10 minutes (varies per model).
• We continually feed the solver with new problems, resulting in it running continually through the day and night.
• We use concurrent solving with 48 threads. After some experimenting i have set the following parameters:

        scip.setRealParam("heuristics/completesol/maxunknownrate", 1)
        scip.setIntParam("heuristics/rens/freq", 20)
        scip.setRealParam("heuristics/rens/minfixingrate", 0.2)
        scip.setIntParam("estimation/restarts/restartlimit", -1)
        scip.setRealParam("estimation/restarts/restartfactor", 5.0)
        scip.setIntParam("parallel/mode", 0)
        scip.setIntParam("parallel/maxnthreads", 48)

• We have been running with 9.2.3 for a while. It crashed around once a day, forcing us to restart the server. The problem was non-reproducible - the same lp file would work fine second time around (which is why I never got around to making an issue for you)
• On friday we bumped to 9.2.4, and based on the changelog we were feeling hopeful that maybe the issue was resolved. It hasn't crashed since.

Issue
When looking at the server today, we noticed that it has stalled - no cpu activity for the past day. The thread in question looks like this:

[166] prio=5 os_prio=0 cpu=11668786.72ms elapsed=248491.56s tid=0x00007fa3e6ef8310 nid=166 runnable  [0x00007f991c4fc000]
java.lang.Thread.State: RUNNABLE
at jscip.SCIPJNIJNI.SCIPsolveConcurrent(Native Method)
at jscip.SCIPJNI.SCIPsolveConcurrent(SCIPJNI.java:250)
at jscip.Scip.solveConcurrent(Scip.java:45)

I know that's probably not super helpful. Unfortunately I don't have much else to give you, other than the background above.

Could it perhaps be the case that the fix mentioned in the changelog of 9.2.4:
"fixed bug with concurrent solve w.r.t. variable indices that led to segmentation faults and fix termination test"
which is what we hoped would fix the crashing, has introduced a regression?

Please tell me if there is anything I can provide you to help with debugging. I can get a lp file perhaps, but given that the same problem has since been solved without issue on the server, I fear it won't really help much.

Technical details

I built scip with:
RUN cmake .. -DTPI=tny -DAUTOBUILD=ON -DGCG=OFF -DUG=OFF -DCMAKE_BUILD_TYPE=Release
If its any help, the server has a AMD EPYC 9454P cpu, is running in linux and in containerized.

Thanks for the work on the library :)

• Lukas

[svigerske]

The fix you mention is in c51fa63. Most of it is about the issue with "variable indices" mentioned in the changelog. The issue mainly came up if one has SOS constraint or SCIP adds symmetry constraints. For the latter, setting misc/usesymmetry = 0 would have been a workaround (if you want to go back to 9.2.3).

The fix to the termination test is only the change to syncstore.c at the very end, I believe (the addition of SCIPgetNLimSolsFound(syncstore->mainscip) > 0). I don't see how this change could be problematic, but you could try whether reverting it helps, or whether reverting the whole c51fa63 helps.

Also CC @pfetsch

[lhoegh]

Thanks, I will try with usesymetry = 0 and report back. Should i expect it to be a lot slower with that setting by the way?

I was actually also looking at
"fixed rare invalid memory access when collecting symmetry information for indicator constraints" and thinking that might be related. We do have some indicator constraints in some of our models.

[svigerske]

Disabling symmetry handling should make things slower in general.

"fixed rare invalid memory access when collecting symmetry information for indicator constraints" was a quite forward fix that just ensured that some arrays were long enough (8bf46c9), it shouldn't introduce a deadlock in concurrent solves.