pointer being freed was not allocated #360
Comments
|
The home-brew binaries are not officially supported by us, and you're unlikely to get much help debugging. Any number of things could be going wrong.
This is not true. You can use any linear solver with the default binaries that we provide by following https://github.com/jump-dev/Ipopt.jl#linear-solvers. |
Did you follow the instructions here: https://jump.dev/JuMP.jl/stable/developers/custom_solver_binaries/? Setting |
I understand that, but in this case the only Julia + Ipopt installation working with additional linear solvers (HSL) is the one that was installed via homebrew. Installing Julia by downloading the official .dmg file (or juliaup) is the one that is causing the segfault, whereas the homebrew installation with HSL solvers works well after
This says I should be able to use one of the following:
I've also tried creating an alias to
I gave Overriding an entire artifact a go just now, but I get the same segftault as soon as I try julia> using Ipopt_jll
julia(9167,0x1fa835b40) malloc: *** error for object 0x117ae47d0: pointer being freed was not allocated
julia(9167,0x1fa835b40) malloc: *** set a breakpoint in malloc_error_break to debug
[9167] signal (6): Abort trap: 6
in expression starting at REPL[2]:1
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 2996 (Pool: 2985; Big: 11); GC: 0
[1] 9166 abort julia --project |
|
I'm confused. Are you trying to use the Is the the thrust of your issue that setting If so, my solution would be: don't use coinbrew to build HSL. Just download the official source code from the HSL website and compile manually on your machine. @amontoison is also working with the HSL folks to provide an official precompiled binaries for Julia that should be available very shortly. The mess of trying to get different binaries to work together is why we're moving away from third party builds wherever possible. I don't know how the homebrew/coinbrew/official Julia builds all interact, but it isn't pretty and the list of things that could be going wrong is long.
It's just the Ipopt option. Line 306 in e5fd613 |
|
@ferrolho If you can wait a little bit, the best solution is to wait the release of JuliaHSL / HSL_jll.jl. |
|
Thank you for your replies, and apologies for only getting back to you now. I tried what Oscar suggested above (i.e. compile everything from source instead of using coinbrew), but without success. At first, I got
However, if I do I noticed that JuliaHSL is now available, so I will give it a try tomorrow. |
|
You shouldn't need to compile Ipopt, only the HSL libraries.
👍 this should mean no more compilation problems. See https://github.com/jump-dev/Ipopt.jl#hsl |
|
Did you try following the installation instructions in the (base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % brew install meson
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % mkdir build
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson setup builddir --buildtype=release --prefix=build
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson setup builddir --buildtype=release --prefix=/Users/oscar/Downloads/coinhsl-archive-2023.05.26/build
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % meson compile -C builddir
... stuff ...
(base) oscar@Oscars-MBP coinhsl-archive-2023.05.26 % julia --project=.
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.6.7 (2022-07-19)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(coinhsl-archive-2023.05.26) pkg> add Ipopt JuMP
# ... stuff ...
(coinhsl-archive-2023.05.26) pkg> st
Status `~/Downloads/coinhsl-archive-2023.05.26/Project.toml`
[b6b21f68] Ipopt v1.4.0
[4076af6c] JuMP v1.11.1
julia> using JuMP, Ipopt
julia> model = Model(Ipopt.Optimizer)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: Ipopt
julia> set_attribute(model, "linear_solver", "ma27")
julia> set_attribute(model, "hsllib", "/Users/oscar/Downloads/coinhsl-archive-2023.05.26/build/lib/libcoinhsl.dylib")
julia> @variable(model, x)
x
julia> @objective(model, Min, x^2)
x²
julia> optimize!(model)
******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit https://github.com/coin-or/Ipopt
******************************************************************************
This is Ipopt version 3.14.4, running with linear solver ma27.
Number of nonzeros in equality constraint Jacobian...: 0
Number of nonzeros in inequality constraint Jacobian.: 0
Number of nonzeros in Lagrangian Hessian.............: 1
Total number of variables............................: 1
variables with only lower bounds: 0
variables with lower and upper bounds: 0
variables with only upper bounds: 0
Total number of equality constraints.................: 0
Total number of inequality constraints...............: 0
inequality constraints with only lower bounds: 0
inequality constraints with lower and upper bounds: 0
inequality constraints with only upper bounds: 0
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 0.0000000e+00 0.00e+00 0.00e+00 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0
Number of Iterations....: 0
(scaled) (unscaled)
Objective...............: 0.0000000000000000e+00 0.0000000000000000e+00
Dual infeasibility......: 0.0000000000000000e+00 0.0000000000000000e+00
Constraint violation....: 0.0000000000000000e+00 0.0000000000000000e+00
Variable bound violation: 0.0000000000000000e+00 0.0000000000000000e+00
Complementarity.........: 0.0000000000000000e+00 0.0000000000000000e+00
Overall NLP error.......: 0.0000000000000000e+00 0.0000000000000000e+00
Number of objective function evaluations = 1
Number of objective gradient evaluations = 1
Number of equality constraint evaluations = 0
Number of inequality constraint evaluations = 0
Number of equality constraint Jacobian evaluations = 0
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations = 0
Total seconds in IPOPT = 0.104
EXIT: Optimal Solution Found. |
Where did you get this error? I highly recommend to download the |
|
Oh, I didn't realise that there was a new way for building the HSL routines! I tried to use The steps I took were very similar to the ones Oscar shared above:
At this point, I was able to use the routines directly with Ipopt in one of two ways:
Thank you for your help. It is nice to know that this more "manual" approach still works. Anyway, I am curious to try JuliaHSL, so I will give that a go soon too. |
I got it when trying to build Ipopt from source with |
|
Nice to hear you got it sorted |
|
I have now tried JuliaHSL, and it also worked really well! 🎉 I have taken the opportunity to edit the instructions in the README.md slightly — see #367. Since I am using a MacBook Air, I had to explicitly delete the quarantine attribute with otherwise, I get the following error: I was then able to run the following test code snippet without issues: import Pkg
Pkg.develop(path="/Users/henrique/Downloads/HSL_jll.jl-2023.5.26")
using JuMP, Ipopt
import HSL_jll
model = Model(Ipopt.Optimizer)
set_attribute(model, "linear_solver", "ma57")
@variable(model, x)
@objective(model, Min, x^2)
optimize!(model)Thank you again both for your help! |
|
Thanks for for your comment @ferrolho. |
|
Did you not need to set the |
Yes, otherwise we get the macOS pop-up that I have shared in my previous message, alongside the following error in the stdout of the Julia session: julia> import HSL_jll
[ Info: Precompiling HSL_jll [017b0a0e-03f4-516a-9b91-836bbd1904dd]
ERROR: InitError: could not load library "/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib"
dlopen(/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib, 0x0001): tried: '/Users/henrique/knitro/lib/libhsl.dylib' (no such file), '/libhsl.dylib' (no such file), '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (code signature in <C8AD7334-EF5B-360F-B0E2-7FA92922192C> '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' not valid for use in process: library load disallowed by system policy), '/System/Volumes/Preboot/Cryptexes/OS/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (no such file), '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' (code signature in <C8AD7334-EF5B-360F-B0E2-7FA92922192C> '/Users/henrique/Downloads/HSL_jll.jl-2023.5.26/override/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib' not valid for use in process: library load disallowed by system policy)
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:116
[3] macro expansion
@ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]
[4] __init__()
@ HSL_jll ~/Downloads/HSL_jll.jl-2023.5.26/src/wrappers/aarch64-apple-darwin-libgfortran5.jl:12
[5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1074
[6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
@ Base ./loading.jl:1020
[7] _tryrequire_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String)
@ Base ./loading.jl:1407
[8] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:1781
[9] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1625
[10] macro expansion
@ ./loading.jl:1613 [inlined]
[11] macro expansion
@ ./lock.jl:267 [inlined]
[12] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1576
during initialization of module HSL_jll |
|
@ferrolho |
That's right. For this approach (importing HSL_jll), I noticed there is no need to set the However, for the other approach (building the HSL routines with Meson), I had to either set the |
Oh, I see what you mean now. I did not try to build it from the source (JuliaHSL-2023.5.26.zip). I suspect it won't be needed, but I can try it out later today and let you know. Just to confirm, that source should be built with Meson too, right? |
|
Thanks! I'm working on Linux so that's quite hard to test it. |
|
I was trying to test this for you, but I am running into an issue trying to compile JuliaHSL from source. Here are the steps I took:
It builds for a while, but it fails near the end at This is with the default and I get the same error. |
|
The issue is that on Mac, the shared libraries for BLAS and LAPACK have more than 10 years. They only updated them recently. Do you have OSX >= 13.3? It's also possible that you have the symbols with an One solution is to link with |
Yes, I am on macOS Ventura 13.4.
These libraries are no longer stored in the filesystem.
So, running /Library/Developer/CommandLineTools/usr/bin/nm: error: /System/Library/Frameworks/Accelerate.framework/Accelerate: No such file or directory
And as for this one, I am getting |
|
Thanks for feedbacks @ferrolho! If you use MacOS Ventura 13.4 it means that you have the updated BLAS and LAPACK but the symbols are without the trailing underscore. gfortran can't find them except if we use For |
|
The moral of this story is that compiling locally is difficult. Just use the pre-built binaries. |
|
Yes, it simplifies a lot of thing for the Julia community. Talking about BLAS and LAPACK symbols with Apple Accelerate. |
I know this question was directed to Oscar, but here are my unsolicited two cents. In addition to saying it is possible to switch dynamically between OpenBLAS, MKL, and AppleAccelerate, it may be equally important to explain why that would be useful (or point to some other online resource where that is explained). As a casual user, I always just use OpenBLAS; but if there are good reasons for using either MKL or AppleAccelerate instead, I would like to know them. |
Yes, the pre-built binaries seem to be the best option. Don't take this as me being stubborn and insisting in compiling from source; I am just trying to answer Alexis' questions since he mentioned he is not able to test things on macOS. I have now been able to compile Here are all the steps to build JuliaHSL from source:
|
|
With that, in this thread, we now have three approaches detailed on a step-by-step basis for using the HSL routines with Ipopt in Julia:
Thank you both for your help in getting all these approaches tested. |
Performance. MKL can be much faster.
If it's simple to document, we should do do. I think we could also add MKL.jl to the tests?
No problem. I think that even though compiling from source works, I'm not going to mention them in the README. It'll result in fewer questions. |
Yes, definitely. |
I suppose it is not possible to use Intel MKL on Apple Silicon. For trying AppleAccelerate with Ipopt, is it as simple as Edit: Oh, wow! It seems that's all is needed, and the performance is very different. On a sample problem of mine, I get - Total seconds in IPOPT = 2.914 # OpenBLAS
+ Total seconds in IPOPT = 1.658 # AppleAccelerate |
Yip. This is pretty expected, and it's why @amontoison has been putting so much work into this. |
Hi! I am using an Apple MacBook Air (M1, 2020), and in my
~/.zshrc, I have a line like thisfor exporting the path to the Ipopt that I built with coinbrew, and which allows me to use different linear solvers besides MA27.
I have been using it without issues, with Julia itself having been installed with
brew install julia(https://github.com/Homebrew/homebrew-core/blob/master/Formula/julia.rb).As you can see in the snippet above, there are no issues loading Ipopt.
However, if I install Julia from the official
.dmgfile or with juliaup, I get this error:I don't know if this is an issue with Ipopt or with Julia, but I decided to post it here first. I had a look at the Homebrew Formula for Julia to try and understand if they are doing something special there for not running into this error, but I am quite out of my depth...
Edit: Oh, and another thing worth noting is that, if I comment out that export line from my
~/.zshrc, this is not an issue.The text was updated successfully, but these errors were encountered: