Failed to generate a custom sysimage for JuMP #3594

hlyang1992 · 2023-12-03T00:22:44Z

According to the documentation Performance Tips,Suggestion 3: use PackageCompiler
, JuMP is compatible with the PackageCompiler.jl package. However, but it fails with the following error message:

ERROR: LoadError: MethodError: no method matching Clarabel.GenPowerConeT(::Vector{Float64}, ::Int64)

The code used to precompile is as follows

PackageCompiler.create_sysimage(
           ["JuMP", "HiGHS"],
           sysimage_path = "customimage." * Libdl.dlext,
           precompile_execution_file = "model.jl",
       )

The text was updated successfully, but these errors were encountered:

joaquimg · 2023-12-03T01:24:29Z

Can you add more detail of what you are trying to do?
Clarabel is another julia package that is not mentioned in your post.

hlyang1992 · 2023-12-03T03:09:28Z

Can you add more detail of what you are trying to do? Clarabel is another julia package that is not mentioned in your post.

I followed the documentation but got the error mentioned above.

Content of model.jl:

using JuMP, HiGHS
model = Model(HiGHS.Optimizer)
set_silent(model)
@variable(model, x >= 0)
@variable(model, 0 <= y <= 3)
@objective(model, Min, 12x + 20y)
@constraint(model, c1, 6x + 8y >= 100)
@constraint(model, c2, 7x + 12y >= 120)
optimize!(model)
open("model.log", "w") do io
    print(io, solution_summary(model; verbose = true))
    return
end

I encountered the error trying to run the following command to generate generate a custom sysimage

using PackageCompiler, Libdl
PackageCompiler.create_sysimage(
    ["JuMP", "HiGHS"],
    sysimage_path = "customimage." * Libdl.dlext,
    precompile_execution_file = "model.jl",
)

odow · 2023-12-03T07:40:36Z

Are you sure? That's the exact code you ran and you're not using Clarabel anywhere?

Can you provide the full log of every command that you ran?

I cannot reproduce your error:

(pc) pkg> st
Status `/private/tmp/pc/Project.toml`
  [87dc4568] HiGHS v1.7.5
  [4076af6c] JuMP v1.16.0
  [9b87118b] PackageCompiler v2.1.15
  [8f399da3] Libdl

julia> using PackageCompiler, Libdl

julia> PackageCompiler.create_sysimage(
           ["JuMP", "HiGHS"],
           sysimage_path = "customimage." * Libdl.dlext,
           precompile_execution_file = "model.jl",
       )
Precompiling project...
  44 dependencies successfully precompiled in 85 seconds
[ Info: PackageCompiler: Executing /private/tmp/pc/model.jl => /var/folders/bg/dzq_hhvx1dxgy6gb5510pxj80000gn/T/jl_packagecompiler_YrqHLh/jl_rjQ25W
[ Info: PackageCompiler: Done
✔ [04m:57s] PackageCompiler: compiling incremental system image

hlyang1992 · 2023-12-03T08:06:33Z

Are you sure? That's the exact code you ran and you're not using Clarabel anywhere?

Can you provide the full log of every command that you ran?

I cannot reproduce your error:

(pc) pkg> st
Status `/private/tmp/pc/Project.toml`
  [87dc4568] HiGHS v1.7.5
  [4076af6c] JuMP v1.16.0
  [9b87118b] PackageCompiler v2.1.15
  [8f399da3] Libdl

julia> using PackageCompiler, Libdl

julia> PackageCompiler.create_sysimage(
           ["JuMP", "HiGHS"],
           sysimage_path = "customimage." * Libdl.dlext,
           precompile_execution_file = "model.jl",
       )
Precompiling project...
  44 dependencies successfully precompiled in 85 seconds
[ Info: PackageCompiler: Executing /private/tmp/pc/model.jl => /var/folders/bg/dzq_hhvx1dxgy6gb5510pxj80000gn/T/jl_packagecompiler_YrqHLh/jl_rjQ25W
[ Info: PackageCompiler: Done
✔ [04m:57s] PackageCompiler: compiling incremental system image

Thank you for your reply, here is the full log

(base) ➜  tmp julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.4 (2023-11-14)
 _/ |\__'_|_|_|\__'_|  |  Built by Homebrew (v1.9.4)
|__/                   |

shell> cat model.jl
using JuMP, HiGHS
model = Model(HiGHS.Optimizer)
set_silent(model)
@variable(model, x >= 0)
@variable(model, 0 <= y <= 3)
@objective(model, Min, 12x + 20y)
@constraint(model, c1, 6x + 8y >= 100)
@constraint(model, c2, 7x + 12y >= 120)
optimize!(model)
open("model.log", "w") do io
    print(io, solution_summary(model; verbose = true))
    return
end


shell> ^C

(@v1.9) pkg> st
Status `~/.julia/environments/v1.9/Project.toml`
  [1e616198] COSMO v0.8.8
  [a076750e] CPLEX v1.0.1
  [336ed68f] CSV v0.10.11
  [9961bab8] Cbc v1.2.0
  [61c947e1] Clarabel v0.6.0
  [e2554f3b] Clp v1.0.3
  [a93c6f00] DataFrames v1.6.1
  [87dc4568] HiGHS v1.7.5
  [7073ff75] IJulia v1.24.2
  [4076af6c] JuMP v1.16.0
  [0327d340] MultiObjectiveAlgorithms v1.3.1
  [9b87118b] PackageCompiler v2.1.15
  [91a5bcdd] Plots v1.39.0
  [c3e4b0f8] Pluto v0.19.32
  [438e738f] PyCall v1.96.2
  [1fd47b50] QuadGK v2.9.1
⌃ [82193955] SCIP v0.11.6
  [276daf66] SpecialFunctions v2.3.1
  [f3b207a7] StatsPlots v0.15.6
  [8f399da3] Libdl
Info Packages marked with ⌃ have new versions available and may be upgradable.

(@v1.9) pkg> ^C

julia> using PackageCompiler, Libdl

julia> PackageCompiler.create_sysimage(
           ["JuMP", "HiGHS"],
           sysimage_path = "customimage." * Libdl.dlext,
           precompile_execution_file = "model.jl",
       )
Precompiling project...
  ✗ Clarabel
  0 dependencies successfully precompiled in 9 seconds. 292 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

Clarabel [61c947e1-3e6d-4ee4-985a-eec8c727bd6e]

Failed to precompile Clarabel [61c947e1-3e6d-4ee4-985a-eec8c727bd6e] to "/Users/hlyang/.julia/compiled/v1.9/Clarabel/jl_VWueix".
ERROR: LoadError: MethodError: no method matching Clarabel.GenPowerConeT(::Vector{Float64}, ::Int64)

Closest candidates are:
  Clarabel.GenPowerConeT(::Vector{Float64}, ::Int32)
   @ Clarabel ~/.julia/packages/Clarabel/YHU2j/src/cones/cone_api.jl:40

Stacktrace:
 [1] __precompile_native()
   @ Clarabel ~/.julia/packages/Clarabel/YHU2j/src/precompile.jl:46
 [2] macro expansion
   @ ~/.julia/packages/Clarabel/YHU2j/src/Clarabel.jl:99 [inlined]
 [3] macro expansion
   @ ~/.julia/packages/SnoopPrecompile/1XXT1/src/SnoopPrecompile.jl:62 [inlined]
 [4] (::Clarabel.var"#65#66")()
   @ Clarabel ~/.julia/packages/Clarabel/YHU2j/src/Clarabel.jl:97
 [5] (::Base.RedirectStdStream)(thunk::Clarabel.var"#65#66", stream::Base.DevNull)
   @ Base ./stream.jl:1429
 [6] top-level scope
   @ ~/.julia/packages/Clarabel/YHU2j/src/Clarabel.jl:97
 [7] include
   @ ./Base.jl:457 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2049
 [9] top-level scope
   @ stdin:3
in expression starting at /Users/hlyang/.julia/packages/Clarabel/YHU2j/src/Clarabel.jl:2
in expression starting at stdin:3
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:1619
 [3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [4] precompile(pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [5] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:171
 [6] precompile()
   @ Pkg.API /usr/local/Cellar/julia/1.9.4/share/julia/stdlib/v1.9/Pkg/src/API.jl:162
 [7] top-level scope
   @ none:1
ERROR: failed process: Process(setenv(`/usr/local/Cellar/julia/1.9.4/bin/julia --color=yes --startup-file=no --pkgimages=no --sysimage=/usr/local/Cellar/julia/1.9.4/lib/julia/sys.dylib -e 'using Pkg; Pkg.precompile()'`,["XPC_FLAGS=0x0", "LSCOLORS=Gxfxcxdxbxegedabagacad", "PATH=/Applications/CPLEX_Studio2211/cplex/bin/x86-64_osx:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/hlyang/opt/miniconda3/bin:/Users/hlyang/opt/miniconda3/condabin", "_CE_M=", "all_proxy=socks5://127.0.0.1:7890", "XPC_SERVICE_NAME=0", "CONDA_PYTHON_EXE=/Users/hlyang/opt/miniconda3/bin/python", "__CF_USER_TEXT_ENCODING=0x1F5:0:0", "TMUX=/private/tmp/tmux-501/default,27087,0", "__CFBundleIdentifier=com.googlecode.iterm2"  …  "_CE_CONDA=", "CONDA_SHLVL=1", "ITERM_SESSION_ID=w0t0p0:5455346F-23A3-41C3-955E-B582A31AA81B", "CONDA_EXE=/Users/hlyang/opt/miniconda3/bin/conda", "HOME=/Users/hlyang", "TERM=screen-256color", "TMUX_PANE=%9", "LS_COLORS=di=1;36:ln=35:so=32:pi=33:ex=31:bd=34;46:cd=34;43:su=30;41:sg=30;46:tw=30;42:ow=30;43", "COLORTERM=truecolor", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] ensurecompiled(project::String, packages::Vector{String}, sysimage::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Na90o/src/PackageCompiler.jl:278
 [5] create_sysimage(packages::Vector{String}; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, julia_init_h_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Na90o/src/PackageCompiler.jl:554
 [6] top-level scope
   @ REPL[4]:1

julia>

odow · 2023-12-03T08:11:06Z

This seems like an issue with Clarabel. I don't know if we can fix in JuMP.

You should instead use a Pkg environment that has the minimal set of packages you need and doesn't include Clarabel.

hlyang1992 · 2023-12-03T08:18:25Z

This seems like an issue with Clarabel. I don't know if we can fix in JuMP.

You should instead use a Pkg environment that has the minimal set of packages you need and doesn't include Clarabel.

Deleting Clarabel worked, thanks.

odow · 2023-12-03T08:19:12Z

I looked into the details. The Int64/Int32 issue is because of
https://github.com/oxfordcontrol/Clarabel.jl/blob/1f654b0b9a0450673fda7e17f95af5fa9798c1c1/src/Clarabel.jl#L6

Built by Homebrew (v1.9.4)

You're not using the official Julia binaries. Please download the official version of Julia from https://julialang.org/downloads/

Other binaries sources are not supported, because they often compile Julia in a way that causes multiple issues with third-party libraries like BLAS.

hlyang1992 · 2023-12-03T11:02:39Z

I looked into the details. The Int64/Int32 issue is because of https://github.com/oxfordcontrol/Clarabel.jl/blob/1f654b0b9a0450673fda7e17f95af5fa9798c1c1/src/Clarabel.jl#L6

Built by Homebrew (v1.9.4)

You're not using the official Julia binaries. Please download the official version of Julia from https://julialang.org/downloads/

Other binaries sources are not supported, because they often compile Julia in a way that causes multiple issues with third-party libraries like BLAS.

I download the official version of Julia from https://julialang.org/downloads/. The following code works fine. This issue should be caused by the Julia version of Homebrew. Thank you.

PackageCompiler.create_sysimage(
           ["JuMP", "HiGHS", "Clarabel"],
           sysimage_path = "customimage." * Libdl.dlext,
           precompile_execution_file = "model.jl",
       )

hlyang1992 closed this as completed Dec 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to generate a custom sysimage for JuMP #3594

Failed to generate a custom sysimage for JuMP #3594

hlyang1992 commented Dec 3, 2023

joaquimg commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023 •

edited

Loading

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

Failed to generate a custom sysimage for JuMP #3594

Failed to generate a custom sysimage for JuMP #3594

Comments

hlyang1992 commented Dec 3, 2023

joaquimg commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023 • edited Loading

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023

hlyang1992 commented Dec 3, 2023

odow commented Dec 3, 2023 •

edited

Loading