Could not load python library when run from compiled julia app on another machine

[sairus7]

When I compile my app that has a PyCall dependency, it could not find python library, because it is compiled with absolute path to that library found on dev machine. So when I run my app on another machine, it can't find it:

fatal: error thrown and no exception handler available.
InitError(mod=:PyCall, error=ErrorException("could not load library "C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll"
The specified module could not be found. "))
jl_errorf at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:77
jl_load_dynamic_library at /cygdrive/c/buildbot/worker/package_win64/build/src\dlload.c:284
#dlopen#3 at .\libdl.jl:117
dlopen at .\libdl.jl:117 [inlined]
init at C:\absolute\path\from\another\machine\.julia\packages\PyCall\7a7w0\src\pyinit.jl:149
jfptr___init___58920.clone_1 at C:\_KTAuto\backend\MainBackendCompiled\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:73

So its this line pyinit.jl:149:

PyCall.jl/src/pyinit.jl

Line 150 in 6b18f38

Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)

[stevengj]

Yes, compiling in the libpython into PyCall was something we did to improve load times (#169). Nowadays, it might be possible to revisit this; see e.g. the PythonCall.jl package for a more dynamic approach.

[sairus7]

There are some other problems compiling app with PythonCall - see here: JuliaPy/PythonCall.jl#146

@stevengj Can you please explain a little bit more, what should be fixed in PyCall to exclude fixed library paths from compiled binaries? Or any other ways to use it with compiled app?

[efJerryYang]

I also encountered this problem today, almost the exact same error message as @sairus7 . At first I used create_sysimage it crashed in a very similar way, so I chose create_app but got this error.

[stevengj]

I wonder if it's as simple as inserting the following line here:

libpython = relpath.(libpython, @__DIR__)

Want to give it a try?

[efJerryYang]

Oh, things seem to crash in the same way with fixed absolute path... Any other operations needed before building the app?

And I wonder if this is the right absolute path for build.jl? I build my app in default julia environment. @stevengj

C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl

[sairus7]

And I wonder if this is the right absolute path for build.jl?

Make sure you dev PyCall in your project environment and make changes into .julia/dev/PyCall/deps/build.jl

[efJerryYang]

Thanks, but this error was thrown when building the pkgs in my project environment

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -y numpy` in root environment
ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)
Closest candidates are:
  relpath(!Matched::String, ::String) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:536
  relpath(!Matched::AbstractString, ::AbstractString) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:573
Stacktrace:
 [1] _broadcast_getindex_evalf
   @ .\broadcast.jl:670 [inlined]
 [2] _broadcast_getindex
   @ .\broadcast.jl:643 [inlined]
 [3] getindex
   @ .\broadcast.jl:597 [inlined]
 [4] copy
   @ .\broadcast.jl:875 [inlined]
 [5] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(relpath), Tuple{Base.RefValue{Ptr{Nothing}}, Base.RefValue{String}}})
   @ Base.Broadcast .\broadcast.jl:860
 [6] top-level scope
   @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:84
 [7] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [8] top-level scope
   @ none:5
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:43

[sairus7]

I build my app in default julia environment.

To compile the project into an app I use project own environment, not the default one.
This is typical script compile_app.sh, notice --project=@. argument:

julia -e 'using Pkg; Pkg.add("PackageCompiler")'

julia --project=@. --startup-file=no -e 'using Pkg; Pkg.instantiate()'

julia --project=@. --startup-file=no -e '
using PackageCompiler;
PackageCompiler.create_app(pwd(), "MyProjectCompiled";
    cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)",
    include_transitive_dependencies=false,
    filter_stdlibs=true,
    precompile_execution_file=["test/runtests.jl"])
'

[sairus7]

ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)

Maybe that's because libpython === nothing?

[efJerryYang]

Sorry, is this solution works for you? And that may be my problem...

[sairus7]

I will check this in a couple of hours

[efJerryYang]

Great, but problem is that the path actually should not be nothing, and after I commented that line pkgs could be built without error (in the Demo environment)

[efJerryYang]

I confirmed that the error was caused by that line of code.

after commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
Precompiling project...
  5 dependencies successfully precompiled in 43 seconds (235 already precompiled)

cancel commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
...

[efJerryYang]

I got the correct build result after replace libpython with libpy_name like this, which is really strange to assign a pointer type with string value.

libpython = relpath.(libpy_name, @__DIR__)

The value of libpy_name is just the fixed absolute path C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll