Ipopt.jl is not thread safe? (segmentation fault was happened.)

[AtsushiSakai]

Hi. first of all, thank you so much for great OSS. I love this OSS.

I am using this Ipopt.jl via JuMP.
When I try to solve multiple optimization problems concurrently using multi-threading function (Base.Threads.@Spawn), segmentation fault was happened.

My environments is:
OS: Ubuntu: Ubuntu 18.04.4 LTS
Julia version: v1.3.0 and v1.4.0
Ipopt.jl version: v0.6.1

This is the console log:

signal (11): Segmentation fault
dmumps_load_MOD_dmumps_190 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_load_MOD_dmumps_190 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_254 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_251 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_244 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_142 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_ at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_f77_ at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_254_ at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_c at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_251_ at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt20MumpsSolverInterface13FactorizationEbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_244 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt20MumpsSolverInterface10MultiSolveEbPKiS2_iPdbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_142 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt16TSymLinearSolver10MultiSolveERKNS_9SymMatrixERSt6vectorINS_8SmartPtrIKNS_6VectorEEESaIS8_EERS4_INS5_IS6_EESaISC_EEbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt18StdAugSystemSolver10MultiSolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRSt6vectorINS_8SmartPtrIS5_EESaISC_EESF_SF_SF_RSA_INSB_IS4_EESaISG_EESJ_SJ_SJ_bi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_f77 at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt15AugSystemSolver5SolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRS5_SA_SA_SA_RS4_SB_SB_SB_bi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
dmumps_c at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt17PDFullSpaceSolver9SolveOnceEbbRKNS_9SymMatrixERKNS_6MatrixES6_S6_S6_S6_S6_RKNS_6VectorES9_S9_S9_S9_S9_S9_S9_S9_S9_ddRKNS_14IteratesVectorERSA at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt20MumpsSolverInterface13FactorizationEbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt17PDFullSpaceSolver5SolveEddRKNS_14IteratesVectorERS1_bb at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt20MumpsSolverInterface10MultiSolveEbPKiS2_iPdbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt21PDSearchDirCalculator22ComputeSearchDirectionEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16TSymLinearSolver10MultiSolveERKNS_9SymMatrixERSt6vectorINS_8SmartPtrIKNS_6VectorEEESaIS8_EERS4_INS5_IS6_EESaISC_EEbi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm22ComputeSearchDirectionEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt18StdAugSystemSolver10MultiSolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRSt6vectorINS_8SmartPtrIS5_EESaISC_EESF_SF_SF_RSA_INSB_IS4_EESaISG_EESJ_SJ_SJ_bi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm8OptimizeEb at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt15AugSystemSolver5SolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRS5_SA_SA_SA_RS4_SB_SB_SB_bi at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication13call_optimizeEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
ZN5Ipopt17PDFullSpaceSolver9SolveOnceEbbRKNS_9SymMatrixERKNS_6MatrixES6_S6_S6_S6_S6_RKNS_6VectorES9_S9_S9_S9_S9_S9_S9_S9_S9_ddRKNS_14IteratesVectorERSA at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEERNS1_INS_16AlgorithmBuilderEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt17PDFullSpaceSolver5SolveEddRKNS_14IteratesVectorERS1_bb at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt21PDSearchDirCalculator22ComputeSearchDirectionEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication12OptimizeTNLPERKNS_8SmartPtrINS_4TNLPEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm22ComputeSearchDirectionEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
IpoptSolve at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm8OptimizeEb at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication13call_optimizeEv at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEERNS1_INS_16AlgorithmBuilderEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
solveProblem at /home/atsushi/.julia/packages/Ipopt/ruIXY/src/Ipopt.jl:342
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication12OptimizeTNLPERKNS_8SmartPtrINS_4TNLPEEE at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
IpoptSolve at /home/atsushi/.julia/packages/Ipopt/ruIXY/deps/usr/lib/libipopt.so (unknown line)
solveProblem at /home/atsushi/.julia/packages/Ipopt/ruIXY/src/Ipopt.jl:342
optimize! at /home/atsushi/.julia/packages/Ipopt/ruIXY/src/MOI_wrapper.jl:914
optimize! at /home/atsushi/.julia/packages/MathOptInterface/XiH8D/src/Bridges/bridge_optimizer.jl:239
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
optimize! at /home/atsushi/.julia/packages/Ipopt/ruIXY/src/MOI_wrapper.jl:914
optimize! at /home/atsushi/.julia/packages/MathOptInterface/XiH8D/src/Bridges/bridge_optimizer.jl:239
optimize! at /home/atsushi/.julia/packages/MathOptInterface/XiH8D/src/Utilities/cachingoptimizer.jl:189
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
optimize! at /home/atsushi/.julia/packages/MathOptInterface/XiH8D/src/Utilities/cachingoptimizer.jl:189
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
optimize!#95 at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:131
#optimize!#95 at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:131
ptimize! at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:107 [inlined]
optimize! at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:107 [inlined]
optimize! at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:107 [inlined]
optimize! at /home/atsushi/.julia/packages/JuMP/CZ8vV/src/optimizer_interface.jl:107 [inlined]
#3 at ./threadingconstructs.jl:126
#3 at ./threadingconstructs.jl:126
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
Allocations: 605632492 (Pool: 605078545; Big: 553947); GC: 1080

My question is that Ipopt.jl is not thread safe? (or Ipopt itself is not thread safe?)

[svigerske]

MUMPS as used by Ipopt is not thread-safe. You may want to try a different linear solver.

[frapac]

Hi AtsushiSakai,
there is also a well known issue concerning thread safety when calling julia callbacks from C/C++ code. See e.g.
jump-dev/JuMP.jl#904
jump-dev/KNITRO.jl#93

Currently, we haven't found a satisfactory workaround.

[AtsushiSakai]

Oh. Thank you guys. I understand this is known issue and there is no workaround.

[frapac]

I think there might exist a workaround. By any chance, would you have a MWE to reproduce your issue with Ipopt.jl?

[AtsushiSakai]

@frapac Thank you!!. This is a MWE.
https://gist.github.com/AtsushiSakai/00a327e3e0b9a5ad7510b708a8e11f21

[frapac]

Looking at your example, I see no reason why it should not work on the Julia side (the problem arises when you are calling Julia code from a multithreaded C application). Your exemple runs on my system. I am using Ipopt compiled with MA27, so I think @svigerske is right and the issue may be related to MUMPS.

[AtsushiSakai]

@frapac Thank you so much for checking it. I got it. I hope MUMPS will be thread safe in the near future : ).

[tamasgal]

Btw. I am running into similar issues, using Ipopt with JuMP. I get segfaults whenever I try to run the optimizer in threads (MUMPS is all over the place in the error).

Can you recommend any similar optimizer which is threadsafe? I see a lot of supported ones already...

[svigerske]

Before you give up on Ipopt...
The Ipopt/Mumps interface of Ipopt 3.14 works around the thread-unsafety issue of Mumps by ensuring that only one thread runs Mumps at a time, so you may be close to single-thread performance, but without the crash.
As others said before, Ipopt supports different linear solvers. The HSL codes and Pardiso from pardiso-project.org are particular good, if you are ok with their license. Building Ipopt with (free) Intel MKL enables the option to use Pardiso from MKL as linear solver. With Ipopt 3.14, an interface to the open-source linear solver from Spral has been added. WSMP is another option.

[tamasgal]

Thanks! I totally overlooked the related issue above and already sent a request for the academic license. I'll try the other solvers, that will make things easier ;)

[AtsushiSakai]

The Ipopt/Mumps interface of Ipopt 3.14 works around the thread-unsafety issue of Mumps by ensuring that only one thread runs Mumps at a time, so you may be close to single-thread performance, but without the crash.

Great improvement!!