Question: How do we use build_shared_lib function Properly ?

[terasakisatoshi]

I'm trying to generate shared library using build_shared_lib which PackageCompiler.jl provides.

Here is my sample code named example.jl

module Example

Base.@ccallable empty()::Cint = 10
Base.@ccallable foo(x::Cint)::Cint = 3
Base.@ccallable function bar(x::Cint)::Cint
    return 3
end

Base.@ccallable function just_print()::Cvoid
    print("Example")
end

Base.@ccallable function bar_segfault(x::Cint)::Cint
    println("Example")
    return 3
end

Base.@ccallable function example(x::Cint)::Cint
    return x
end
end

Apparently building shared object succeed by using the following the script named gen_lib.jl:

using PackageCompiler
build_shared_lib("example.jl")

$ julia gen_lib.jl

This will create builddir/example.dylib.

The problem is some function works, but other cause segmentation fault.

Here is a code to reproduce our problem:

• environment

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.1 (2019-05-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc (2019-05-16 04:10 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin15.6.0)
  CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

• make call script named call_example.jl

const dylibpath="builddir/example"
@show ccall((:foo,dylibpath),Cint,(Cint,),100)
@show ccall((:bar,dylibpath),Cint,(Cint,),100)

println("next :bar_segfauld will be seg fault...")

@show ccall((:bar_segfault,dylibpath),Cint,(Cint,),100)

$ julia call_example.jl
ccall((:foo, dylibpath), Cint, (Cint,), 100) = 3
ccall((:bar, dylibpath), Cint, (Cint,), 100) = 3
next :bar_segfauld will be seg fault...

signal (11): Segmentation fault: 11
in expression starting at /Users/terasaki/tmp/sample/call_example.jl:7
unknown function (ip: 0xffffffffffffffff)
Allocations: 295934 (Pool: 295852; Big: 82); GC: 0
Segmentation fault: 11

How do we use shared object Properly ?

[yatra9]

you may want to use build_shared_lib("example.jl", init_shared=true) and call init_jl_runtime() before calling ccall((:bar_segfault,dylibpath),Cint,(Cint,),100).

see #122

[terasakisatoshi]

@yatra9

Hi, thank you for your suggestion.

I modified my script with adding keyword option init_shared=true i.e.

# gen_shared_lib.jl
using PackageCompiler

build_shared_lib("example.jl", init_shared=true)

But, the following script also occurs segmentation fault: 11. This phenomena happens on my another Linux PC (Ubuntu 16.04 its Julia version is 1.1.1.)

# call_example.jl
const dylibpath="builddir/example.dylib"
@show ccall((:empty,dylibpath),Cint,(Ptr{Cvoid},),C_NULL)

@info "call init_jl_runtime"
@show ccall((:init_jl_runtime,dylibpath),Cvoid,(Ptr{Cvoid},),C_NULL)

@info "try to call bar_segfault"

@show ccall((:bar_segfault,dylibpath),Cint,(Cint,),Cint(100))

$ julia call_example.jl
ccall((:empty, dylibpath), Cint, (Ptr{Cvoid},), C_NULL) = 10
[ Info: call init_jl_runtime
ccall((:init_jl_runtime, dylibpath), Cvoid, (Ptr{Cvoid},), C_NULL) = nothing
[ Info: try to call bar_segfault

signal (11): Segmentation fault: 11
in expression starting at /Users/terasaki/work/juliaExer/embeddingJulia/packageCompiler/call_example.jl:9
unknown function (ip: 0xffffffffffffffff)
Allocations: 1068821 (Pool: 1068636; Big: 185); GC: 1
Segmentation fault: 11

Did you make it ?

[yatra9]

umm...calling julia from julia causes somthing wrong?
isn't jl_init reentrant?

[kdheepak]

I'm also encountering the same issue on my MacOS Sierra 10.12.6.

[kdheepak]

I figured this out. I'll submit a PR for this fix shortly. Thanks to @pfitzseb as well.

[kdheepak]

@terasakisatoshi can you try my branch from the PR referenced in this issue? On my mac it works fine. But on Linux (Ubuntu), I'm not able to compile anything.

parallels@parallels-vm:~/gitrepos/PackageCompiler.jl$ julia --project juliac.jl -vasij `pwd`/hello.jl
Julia program file:
  "/home/parallels/gitrepos/PackageCompiler.jl/hello.jl"
Build directory:
  "/home/parallels/gitrepos/PackageCompiler.jl/builddir"
Make build directory
Build static library "hello.a":
  atexit_hook_copy = copy(Base.atexit_hooks) # make backup
# clean state so that any package we use can carelessly call atexit
empty!(Base.atexit_hooks)
Base.__init__()
Sys.__init__() #fix https://github.com/JuliaLang/julia/issues/30479
using REPL
Base.REPL_MODULE_REF[] = REPL
Mod = @eval module $(gensym("anon_module")) end
# Include into anonymous module to not polute namespace
Mod.include("/home/parallels/gitrepos/PackageCompiler.jl/hello.jl")
Base._atexit() # run all exit hooks we registered during precompile
empty!(Base.atexit_hooks) # don't serialize the exit hooks we run + added
# atexit_hook_copy should be empty, but who knows what base will do in the future
append!(Base.atexit_hooks, atexit_hook_copy)

ERROR: LoadError: failed process: Process(`/home/parallels/julia-1.1.0/bin/julia --output-o=hello.a --track-allocation=none --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --project=@. --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/home/parallels/julia-1.1.0/lib/julia/sys.so --compiled-modules=yes --optimize=2 /home/parallels/gitrepos/PackageCompiler.jl/sysimg/run_julia_code.jl`, ProcessSignaled(9)) [0]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:785 [inlined]
 [3] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
 [4] run at ./process.jl:724 [inlined]
 [5] #run_julia#1 at /home/parallels/gitrepos/PackageCompiler.jl/src/compiler_flags.jl:225 [inlined]
 [6] #run_julia at ./none:0 [inlined]
 [7] (::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Nothing,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),NTuple{14,Nothing}}},String})() at /home/parallels/gitrepos/PackageCompiler.jl/src/static_julia.jl:263
 [8] cd(::getfield(PackageCompiler, Symbol("##13#14")){Base.Iterators.Pairs{Symbol,Nothing,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),NTuple{14,Nothing}}},String}, ::String) at ./file.jl:96
 [9] #build_object#12(::Base.Iterators.Pairs{Symbol,Nothing,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),NTuple{14,Nothing}}}, ::Function, ::String, ::String, ::String, ::Bool) at /home/parallels/gitrepos/PackageCompiler.jl/src/static_julia.jl:262
 [10] #build_object at ./none:0 [inlined]
 [11] build_object(::String, ::String, ::String, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing) at /home/parallels/gitrepos/PackageCompiler.jl/src/static_julia.jl:241
 [12] #static_julia#5(::Nothing, ::Bool, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Array{String,1}, ::typeof(static_julia), ::String) at /home/parallels/gitrepos/PackageCompiler.jl/src/static_julia.jl:162
 [13] (::getfield(PackageCompiler, Symbol("#kw##static_julia")))(::NamedTuple{(:init_shared, :autodeps, :verbose, :copy_julialibs, :shared, :cc_flags),Tuple{Bool,Bool,Bool,Bool,Bool,Array{String,1}}}, ::typeof(static_julia), ::String) at ./none:0
 [14] julia_main(::Array{String,1}) at /home/parallels/gitrepos/PackageCompiler.jl/juliac.jl:180
 [15] top-level scope at /home/parallels/gitrepos/PackageCompiler.jl/juliac.jl:186
 [16] include at ./boot.jl:326 [inlined]
 [17] include_relative(::Module, ::String) at ./loading.jl:1038
 [18] include(::Module, ::String) at ./sysimg.jl:29
 [19] exec_options(::Base.JLOptions) at ./client.jl:267
 [20] _start() at ./client.jl:436
in expression starting at /home/parallels/gitrepos/PackageCompiler.jl/juliac.jl:185

There's no error being printed out. If I run the command that failed as a separate process, it just says Killed. Maybe I'm running out of memory on my virtual machine?

parallels@parallels-vm:~/gitrepos/PackageCompiler.jl$ /home/parallels/julia-1.1.0/bin/julia --output-o=hello.a --track-allocation=none --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --project=@. --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/home/parallels/julia-1.1.0/lib/julia/sys.so --compiled-modules=yes --optimize=2 /home/parallels/gitrepos/PackageCompiler.jl/sysimg/run_julia_code.jl
Killed
parallels@parallels-vm:~/gitrepos/PackageCompiler.jl$ 

I'm curious to see if this happens on yours as well. I'm assuming not.

[terasakisatoshi]

Hi, @kdheepak Thank you for your work.

• I've tried your PR and found that it works fine for my mac.

Also, I've tried it on my Ubuntu machine 16.04.
Build works fine

julia --project juliac.jl -vasij `pwd`/hello.jl

Julia program file:
  "/home/terasaki/tmp/PackageCompiler.jl/hello.jl"
Build directory:
  "/home/terasaki/tmp/PackageCompiler.jl/builddir"
Make build directory
Build static library "hello.a":
  atexit_hook_copy = copy(Base.atexit_hooks) # make backup
# clean state so that any package we use can carelessly call atexit
empty!(Base.atexit_hooks)
Base.__init__()
Sys.__init__() #fix https://github.com/JuliaLang/julia/issues/30479
using REPL
Base.REPL_MODULE_REF[] = REPL
Mod = @eval module $(gensym("anon_module")) end
# Include into anonymous module to not polute namespace
Mod.include("/home/terasaki/tmp/PackageCompiler.jl/hello.jl")
Base._atexit() # run all exit hooks we registered during precompile
empty!(Base.atexit_hooks) # don't serialize the exit hooks we run + added
# atexit_hook_copy should be empty, but who knows what base will do in the future
append!(Base.atexit_hooks, atexit_hook_copy)


Build shared library "hello.so":
  `cc -shared '-DJULIAC_PROGRAM_LIBNAME="/home/terasaki/tmp/PackageCompiler.jl/builddir/hello.so"' -o hello.so -Wl,--whole-archive hello.a -Wl,--no-whole-archive /home/terasaki/tmp/PackageCompiler.jl/builddir/lib_init.c -std=gnu99 -I/home/terasaki/julia_dev/include/julia -DJULIA_ENABLE_THREADING=1 -fPIC -L/home/terasaki/julia_dev/lib -Wl,--export-dynamic -Wl,-rpath,/home/terasaki/julia_dev/lib -Wl,-rpath,/home/terasaki/julia_dev/lib/julia -ljulia -m64`
Copy Julia libraries to build directory:
  libjulia.so
  libjulia.so.1
  libjulia.so.1.3
  libLLVM-6.0.1.so
  libLLVM-6.0.so
  libLLVM.so
  libamd.so
  libamd.so.2
  libamd.so.2.4.6
  libcamd.so
  libcamd.so.2
  libcamd.so.2.4.6
  libccalltest.so
  libccolamd.so
  libccolamd.so.2
  libccolamd.so.2.9.6
  libcholmod.so
  libcholmod.so.3
  libcholmod.so.3.0.13
  libcolamd.so
  libcolamd.so.2
  libcolamd.so.2.9.6
  libcurl.so
  libcurl.so.4
  libcurl.so.4.5.0
  libdSFMT.so
  libgcc_s.so.1
  libgfortran.so.3
  libgit2.so
  libgit2.so.0.27.7
  libgit2.so.27
  libgmp.so
  libgmp.so.10
  libgmp.so.10.3.2
  libllvmcalltest.so
  libmbedcrypto.so
  libmbedcrypto.so.2.16.0
  libmbedcrypto.so.3
  libmbedtls.so
  libmbedtls.so.12
  libmbedtls.so.2.16.0
  libmbedx509.so
  libmbedx509.so.0
  libmbedx509.so.2.16.0
  libmkl_rt.so
  libmpfr.so
  libmpfr.so.6
  libmpfr.so.6.0.2
  libopenblas64_.so
  libopenblas64_.so.0
  libopenlibm.so
  libopenlibm.so.2
  libopenlibm.so.2.5
  libpcre2-8.so
  libpcre2-8.so.0
  libpcre2-8.so.0.7.0
  libquadmath.so.0
  libspqr.so
  libspqr.so.2
  libspqr.so.2.0.9
  libssh2.so
  libssh2.so.1
  libssh2.so.1.0.1
  libsuitesparse_wrapper.so
  libsuitesparseconfig.so
  libsuitesparseconfig.so.5
  libsuitesparseconfig.so.5.4.0
  libumfpack.so
  libumfpack.so.5
  libumfpack.so.5.7.8
All done

But, call hello.so from python fails...

Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> lib=ctypes.CDLL("./builddir/hello.so")
fatal: error thrown and no exception handler available.
ErrorException("could not load symbol "jl_n_threads":
python: undefined symbol: jl_n_threads")
rec_backtrace at /home/terasaki/work/jlcross/mac/julia/src/stackwalk.c:94
record_backtrace at /home/terasaki/work/jlcross/mac/julia/src/task.c:217 [inlined]
jl_throw at /home/terasaki/work/jlcross/mac/julia/src/task.c:454
jl_errorf at /home/terasaki/work/jlcross/mac/julia/src/rtutils.c:77
jl_dlsym at /home/terasaki/work/jlcross/mac/julia/src/dlload.c:234
jl_load_and_lookup at /home/terasaki/work/jlcross/mac/julia/src/runtime_ccall.cpp:63
nthreads at ./threadingconstructs.jl:19 [inlined]
__preinit_threads__ at ./task.jl:425
_jl_invoke at /home/terasaki/work/jlcross/mac/julia/src/gf.c:2043 [inlined]
jl_apply_generic at /home/terasaki/work/jlcross/mac/julia/src/gf.c:2213
jl_apply at /home/terasaki/work/jlcross/mac/julia/src/julia.h:1630 [inlined]
_julia_init at /home/terasaki/work/jlcross/mac/julia/src/init.c:804
jl_init_with_image__threading at /home/terasaki/work/jlcross/mac/julia/src/jlapi.c:54
init_jl_runtime at ./builddir/hello.so (unknown line)
initialize at ./builddir/hello.so (unknown line)
unknown function (ip: 0x7f0f2adec6c9)
unknown function (ip: 0x7f0f2adec7da)
unknown function (ip: 0x7f0f2adf18f1)
unknown function (ip: 0x7f0f2adec573)
unknown function (ip: 0x7f0f2adf0db8)
dlopen_doit at /build/glibc-LK5gWL/glibc-2.23/dlfcn/dlopen.c:66
unknown function (ip: 0x7f0f2adec573)
_dlerror_run at /build/glibc-LK5gWL/glibc-2.23/dlfcn/dlerror.c:163
__dlopen at /build/glibc-LK5gWL/glibc-2.23/dlfcn/dlopen.c:87
warning: parsing line table prologue at 0x00000000 should have ended at 0x000004a1 but it ended at 0x000001e6
py_dl_open at (null):0
_PyCFunction_FastCallDict at python (unknown line)
unknown function (ip: 0x5619f4bb7c1b)
_PyEval_EvalFrameDefault at python (unknown line)
unknown function (ip: 0x5619f4bb142d)
_PyFunction_FastCallDict at python (unknown line)
_PyObject_FastCallDict at python (unknown line)
_PyObject_Call_Prepend at python (unknown line)
PyObject_Call at python (unknown line)
unknown function (ip: 0x5619f4b8b78a)
unknown function (ip: 0x5619f4bb7f56)
_PyObject_FastCallDict at python (unknown line)
unknown function (ip: 0x5619f4bb7d6d)
_PyEval_EvalFrameDefault at python (unknown line)
PyEval_EvalCodeEx at python (unknown line)
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x5619f4c34073)
unknown function (ip: 0x5619f4afacf3)
PyRun_InteractiveLoopFlags at python (unknown line)
unknown function (ip: 0x5619f4afaf49)
unknown function (ip: 0x5619f4afd1f6)
main at python (unknown line)
__libc_start_main at /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291
unknown function (ip: 0x5619f4be11a7)

[kdheepak]

And you are using Julia Version 1.1.1? Can you open a new Julia REPL and run Int(unsafe_load(cglobal(:jl_n_threads, Cint)))?

[kdheepak]

Also, can you run nm ./builddir/libjulia.so | grep 'jl_n_threads' from the directory where you run julia --project juliac.jl -vasij pwd/hello.jl?

You should get a symbol that points to a memory address as seen in the screenshot above.

Also, just to clarify, the reason I'm running julia --project juliac.jl -vasij `pwd`/hello.jl is because I am in PackageCompiler.jl git repository. So I'm initializing its project environment and then running juliac.jl. In order to do this, I also add to add ArgParse.jl to this or the root environment. Additionally, I had to run build in the Pkg REPL mode to then allow using PackageCompiler to work. I'm sure you've done all this or something similar, but just wanted to make sure we are on the same page :)

PackageCompiler does not work for me on my Ubuntu installation, but this is what I get when i run the above commands on my installed version of the Julia runtime.

[kdheepak]

Also, did you build Julia from source?

[kdheepak]

Lastly, can you try one other thing?

Create a file called test.c:

// test.c
#include <julia.h>
JULIA_DEFINE_FAST_TLS() // only define this once, in an executable (not in a shared library) if you want fast code.

int main(int argc, char *argv[])
{
    /* required: setup the Julia context */
    jl_init__threading();

    /* run Julia commands */
    jl_eval_string("println(sqrt(2.0))");

    /* strongly recommended: notify Julia that the
         program is about to terminate. this allows
         Julia time to cleanup pending write requests
         and run all finalizers
    */
    jl_atexit_hook(0);
    return 0;
}

Set JULIA_DIR environment variable to point to the folder of your Julia installation:

export JULIA_DIR="/Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia"

And run the following:

gcc -o output test.c -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib/julia -L$JULIA_DIR/lib -ljulia -Wl,-rpath -Wl,$JULIA_DIR/lib -Wl,-rpath -Wl,$JULIA_DIR/lib/julia

Then run output

./output

And let me know what happens.

[kdheepak]

Oddly enough, this embedding example above works for me on Linux too. But when I try to load the library from Python I get the same segfault that @pfitzseb gets:

[kdheepak]

Interesting, nearly same error on my mac:

python test.py
fatal: error thrown and no exception handler available.
#<null>
rec_backtrace at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_throw at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_errorf at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_dlsym at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_load_and_lookup at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jlplt_jl_cpu_threads_13841.clone_1 at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
__init__ at ./sysinfo.jl:108
jl_module_run_initializer at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
_julia_init at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_init__threading at /Users/$USER/Applications/Julia-1.1.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
ffi_call_unix64 at /Users/$USER/miniconda3/lib/libffi.6.dylib (unknown line)
ffi_call at /Users/$USER/miniconda3/lib/libffi.6.dylib (unknown line)
_ctypes_callproc at /Users/$USER/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so (unknown line)
PyCFuncPtr_call at /Users/$USER/miniconda3/lib/python3.7/lib-dynload/_ctypes.cpython-37m-darwin.so (unknown line)
_PyObject_FastCallKeywords at /Users/$USER/miniconda3/bin/python (unknown line)
call_function at /Users/$USER/miniconda3/bin/python (unknown line)
_PyEval_EvalFrameDefault at /Users/$USER/miniconda3/bin/python (unknown line)
_PyEval_EvalCodeWithName at /Users/$USER/miniconda3/bin/python (unknown line)
PyRun_FileExFlags at /Users/$USER/miniconda3/bin/python (unknown line)
PyRun_SimpleFileExFlags at /Users/$USER/miniconda3/bin/python (unknown line)
pymain_main at /Users/$USER/miniconda3/bin/python (unknown line)
main at /Users/$USER/miniconda3/bin/python (unknown line)

[kdheepak]

Never mind my last two comments, they work on Linux and Mac if you use RTLD_GLOBAL.