PackageCompiler will not compile my (simple) Julia program to a static binary due to ArgParse errors

[isene]

I am trying to compile my program to an executable on Linux using PackageCompiler - but can't get it to work on either Julia 0.6 or 0.7. In both cases it's ArgParse that creates the error, but different errors for the two versions. See the attached files for detail (I renamed my program to be compiled here from O6.jl to O6.txt due to limitations of uploading .jl files)

O6.txt
julia06.txt
julia07.txt

[NHDaly]

Ah, this one is easy! The problem isn't related to ArgParse at all, your julia program is simply missing julia_main, which is required in order to be compiled. Adding this to the bottom of your file O6.txt is enough to fix it (for julia v0.6):

Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
  O6()
  return 0
end

I've just noticed that's not mentioned anywhere on the README at all... I will send a PR up to fix that now.

EDIT: oops, sorry, I hit "shift-enter" which apparently submits the comment, so it was submitted before I finished typing. Sorry!

[NHDaly]

(You can see the problem here, in the output from juliac):

...
/tmp/ccVlecLr.o: In function `main':
program.c:(.text+0x118): undefined reference to `julia_main'
collect2: error: ld returned 1 exit status
ERROR: LoadError: failed process: Process(`cc '-DJULIAC_PROGRAM_LIBNAME="O6.so"' -o /home/geir/Dropbox/G/Tech/Julia/builddir/O6 /home/geir/.julia/v0.6/PackageCompiler/examples/program.c /home/geir/Dropbox/G/Tech/Julia/builddir/O6.so -m64 -std=gnu99 -I/home/geir/Dropbox/Dl/tmp/julia-06/include/julia -DJULIA_ENABLE_THREADING=1 -fPIC -L/home/geir/Dropbox/Dl/tmp/julia-06/lib -Wl,--export-dynamic -Wl,-rpath,/home/geir/Dropbox/Dl/tmp/julia-06/lib -Wl,-rpath,/home/geir/Dropbox/Dl/tmp/julia-06/lib/julia -ljulia '-Wl,-rpath,$ORIGIN'`, ProcessExited(1)) [1]
...

But there's a lot of output, so that message is hard to find, and it's also not clear what to do to fix it.

[lucatrv]

I assumed there was an issue with ArgParse because I could not statically compile juliac.jl (but had never really bothered to better analyze this issue before...) now I found out that the issue is actually due to using PackageCompiler and to calling julia_main(ARGS) within juliac.jl.
It appears to me that when Julia statically compiles a script with the command $julia_cmd --output-o $o_file -e $expr, it also executes it. This can easily be verified by compiling a simple script with just one line println("test"): "test" is shown while compiling.
Now I think this is why juliac.jl cannot be statically compiled, and maybe it is a Julia bug, do you agree?

[NHDaly]

Hmm, no I don't think so! I think it needs to execute the script, otherwise it might not know which code to compile!

Take the following trivial example:

if VERSION >= v"0.7"
  include("mycode-0.7")
else
  include("mycode-0.6")
end

---

I have been thinking about global, "script-like" code as basically static, compile-time only logic, similar to c++ preprocessor macros, but way better because it's just normal, first class julia code.

[isene]

Thanks. Adding that julia_main at the end there got me an executable. And it does run and gives the correct result. But it also gives this warning every time I run it (how do I remove this?):

WARNING: redefining constant JULIA_HOME

Also, being new to Julia, this looks ugly: julia_main(ARGS::Vector{String})::Cint
Why is all that needed?

[lucatrv]

@NHDaly one option could be to be able to detect programmatically if a script is being statically compiled, and based on that either do or do not execute parts of code.
At present I am not able either to check if the script is being statically compiled, or if it has been statically compiled, get(ENV, "COMPILE_STATIC", "false") == "false" does not seem to work (see juliac.jl).

[NHDaly]

Thanks. Adding that julia_main at the end there got me an executable. And it does run and gives the correct result.

Yay! Glad to hear it.

---

WARNING: redefining constant JULIA_HOME

This is basically just some weirdness in julia v0.6. The next version,v0.7 should be officially, stably released in around a month or less, and then the WARNING will disappear. Sorry if it's annoying... Here is the original context, if you're interested (it was hairy!):
#47
#55

If it bugs you, would you open an Issue about it? We could maybe look more into how to silence the warning in the meantime...

---

Also, being new to Julia, this looks ugly: julia_main(ARGS::Vector{String})::Cint
Why is all that needed?

Yes, I agree, it is a bit ugly. The answer here is more or less that it's ugly because that function is being exposed as an interface that is callable from C, so it needs more stuff than a normal julia function.

More precisely, the Base.@ccallable macro causes julia to emit a non-mangled, C ABI compatible function. So the function declaration in julia needs to be explicit about all the same things a function in C would be explicit about, namely: A. the name of the function (julia_main), B. the name of each parameter (ARGS) and C. the type of each parameter (Vector{String}), and D. the return type (Cint). (Cint is the same as int in C. You could also use Int64/Int32 instead depending on your system.)

This is what an effectively similar function would look like in C:

int julia_main(char* ARGS[]) { ... }

You can see all the same parts are there (except that C doesn't have Vectors, just arrays, and no strings, just pointers to characters), they're just in a different order: D. int A. julia_main B. ARGS C. char*[]

In Julia, it's optional to declare the types of function parameters and return types. And in fact, it's often better not to. For example, the following function will work for any type whose values can be added together and divided by a float, and that's just not a thing that's possible to write in C:

julia> average(a,b) = (a + b) / 2.0
julia> average(0.5, 1+2im)  # here I'm averaging a float and a complex number. The author of average might not have even _known_ about complex numbers.
>> 0.75 + 1.0im

But sometimes there are reasons to specify a type, such as specializing a function's behavior based on the types of its inputs. C interoperability is another such case. :) You can read more here if you're interested:
https://docs.julialang.org/en/stable/manual/calling-c-and-fortran-code/index.html

---

Finally, if you want, you can simply omit the parameters if your program isn't going to use them and just declare it as julia_main()::Cint.

[NHDaly]

@lucatrv Yeah I saw that as well. Actually, you can fix that by calling build_executable through withenv to set "COMPILE_STATIC" to true! I don't know how that code got there in the first place though..?

I've opened an issue specific to what you're trying to do here: #78 Let's talk more about it there! :)

[NHDaly]

@isene: Also, yay! welcome to julia!! :)

[isene]

Amazing comment. Thanks for taking the time to explain this so thoroughly.

[lucatrv]

Can this issue be closed?

[blaiseli]

@lucatrv May I reactivate this not-yet-closed issue? The title corresponds to my initial issue.

I had a program containing a julia_main function, but this function was then called, which caused errors with ArgParse in the message.

Thanks to the above explanations, I understand now that my program should just define julia_main, but not execute it, and I imagine that execution will be handled by some C code somewhere.

So I commented out the call to julia_main.

Now, with julia_main still being defined, but not called, running build_executable on my program surprisingly results in the following error:

Julia program file:
  "/home/bli/src/qaf_demux/Julia/QafDemux/bin/qaf_demux.jl"
C program file:
  "/home/bli/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
Build directory:
  "/home/bli/src/qaf_demux/Julia/QafDemux/builddir"
/tmp/ccAAtpnB.o: In function `main':
program.c:(.text+0x153): undefined reference to `julia_main'
collect2: error: ld returned 1 exit status
ERROR: LoadError: failed process: Process(`cc '-DJULIAC_PROGRAM_LIBNAME="qaf_demux.so"' -o qaf_demux /home/bli/.julia/packages/PackageCompiler/CJQcs/examples/program.c qaf_demux.so -std=gnu99 -I/home/bli/src/julia/usr/include/julia -DJULIA_ENABLE_THREADING=1 -fPIC -L/home/bli/src/julia/usr/lib -Wl,--export-dynamic -Wl,-rpath,/home/bli/src/julia/usr/lib -Wl,-rpath,/home/bli/src/julia/usr/lib/julia -ljulia -m64 '-Wl,-rpath,$ORIGIN'`, ProcessExited(1)) [1]

Stacktrace:

etc.

I double-checked that julia_main is defined in my code:

$ grep "julia_main" /home/bli/src/qaf_demux/Julia/QafDemux/bin/qaf_demux.jl
function julia_main()::Cint
# julia_main()

Here is my build script:

$ cat deps/build.jl
import Pkg
Pkg.add("ArgParse")
Pkg.add("IterTools")
Pkg.add("FASTX")
Pkg.add("CodecZlib")
Pkg.add("PackageCompiler")
using PackageCompiler
build_executable(joinpath(@__DIR__, "../bin/qaf_demux.jl"), "qaf_demux")

(By the way, is that a correct way to write a build script? I couldn't find documentation giving examples of what kind of things should be put in a build script, just documentation saying that a project can have one in deps/build.jl, and that it will be executed upon Pkg.build().)

And I execute it as follows:

$ julia deps/build.jl

All the above is done from the QafDemux directory, and I'm running julia 1.2:

$ julia --version
julia version 1.2.0

[blaiseli]

Actually, the julia_main function needs a more complete signature. It works with Base.@ccallable function julia_main(ARGS::Vector{String})::Cint.

Now my application compiles, and runs with its --help option, but crashes when actually trying to process data (which happens OK with the non-compiled julia script with a normal julia_main). Any help appreciated, here or at https://discourse.julialang.org/t/understanding-runtime-errors-with-packagecompiler-built-executables/29120