Creating a binary that links against Julia

[fingolfin]

For various reasons, I need to create a binary build for a binary that links against Julia resp. libjulia (to interact with the Julia GC, but that should be irrelevant). This requires Julia >= 1.2

My problem is that in the build environment, there is no Julia installed, and in particular no Julia headers, so I can't do that. Is there any way around this? I realize this is probably a relatively rare thing to do, so perhaps BinaryBuilder does not cover it yet? If it does not, is there are at least principle interest in supporting this? (If so, I'd be inclined to try and help implement it).

Not sure if it is relevant, but note that I am currently using BinaryBuilder 0.1.4, because that's the version being shipped. But I see that there are tons of changes in this repository compared to that, which makes me wonder what I should be using.

[giordano]

You'd probably need to use Julia as dependency. There is a builder in Yggdrasil, but it probably works for a handful of platforms and it's for Julia 1.0.3.

In general, if you're looking for a builder you should checkout Yggdrasil, and contribute there a new one

[fingolfin]

That Julia builder looks like it builds another copy of Julia, though? That's not sufficient; it is vital to link against the same Julia that is loading "my" binary.

Yggdrasil looks like it doesn't play with the current release of BinaryBuilder (0.1.4) though, or does it?

[giordano]

Yggdrasil looks like it doesn't play with the current release of BinaryBuilder (0.1.4) though, or does it?

Yeah, sorry, forgot to reply on this. Yggdrasil is using master version of BinaryBuilder.jl, which requires Julia 1.3-rc4 or later

[barche]

That Julia builder looks like it builds another copy of Julia, though? That's not sufficient; it is vital to link against the same Julia that is loading "my" binary.

Why is that, exactly? We're using those binaries to build the C++ part of CxxWrap, and since it is using dynamic linking it works fine with libjulia 1.0 - 1.3, even if at compile time only the 1.0 version was available.

[fingolfin]

Sorry, with "link" I was referring to "dynamic link at runtime", not "creating the binary during the build stage".

I didn't realize that I could use that "Julia builder" and then later on link against the actual "main" Julia. If that's possible, then this does again sound like a potentially viable venue -- thank you for the hint! I'll take a look at how CxxWrap does it.

However, we do use APIs that only were added in Julia 1.1 (via a PR by us), so we need that at the latest. We also access relatively low-level parts of the GC directly, so the question is whether that part of the ABI stayed compatible from Julia 1.1 to 1.2 resp. 1.3.

[fingolfin]

All in all, I am still very much confused what version of BinaryBuilder.jl to target. The one in master seems to be radically different from the 0.1.4 release; at the same time, it also seems to be heavily in flux? Should I still use it?

[giordano]

All in all, I am still very much confused what version of BinaryBuilder.jl to target. The one in master seems to be radically different from the 0.1.4 release;

The latest stable version is very old and almost no one is using it. The code on master is very much different from that version, but the only big differences in terms of builders are how to specify products (they're now a Vector, instead of a function returning a Vector) and dependencies (just use the name of a JLL package, instead of the URL of a remote build.jl script).

at the same time, it also seems to be heavily in flux?

Most of the new development happened in PR #441, but after that PR has been merged the code has been quite stable. Many of the changes that we had since then are about tweaking CI in Yggdrasil and don't affect users.

Should I still use it?

Yes, as long as you're willing to use Julia 1.3.0-rc4 or later.

[barche]

However, we do use APIs that only were added in Julia 1.1 (via a PR by us), so we need that at the latest. We also access relatively low-level parts of the GC directly, so the question is whether that part of the ABI stayed compatible from Julia 1.1 to 1.2 resp. 1.3.

Yes, updating the builder to get all the new point releases in should fix that. I'm kind of surprised we didn't hit problems yet with CxxWrap, so I'd probably also use the updated versions when available just to be safe.

[fingolfin]

Is there a way to specify multiple versions in a builder? Or does this require duplicating the builder (possible with an include to reduce code duplication)?

I noticed that the Julia builder has lots of commented out dependencies. Wonder what the story is there...

[barche]

I think the best way would be to sequentially generate new versions of the builder, the JLL packages that are generated are versioned and can be referred to by version in the build_tarballs.jl dependencies section. The version is also in the Yggdrasil release download path, so you could craft a legacy script for Julia versions before 1.3 that still use the binaries built using the artifacts system.

A nice overview of the new system is here: https://julialang.org/blog/2019/11/artifacts

[fingolfin]

Ok, that sounds interesting - but is it limited to Julia 1.3, or available in 1.1 & 1.2?

[barche]

As far as I understand, you can build the 1.1 and 1.2 binaries on Yggdrasil, but you will need to manually add a build.jl to download them on those versions, since it seems the new BinaryBuilder doesn't generate the build.jl anymore. The generated files end up in a repository in https://github.com/JuliaBinaryWrappers

[fingolfin]

Note that I actually need this for specific Julia versions; experimentally, binaries using some of the lower level Julia "APIs" (they are not really official APIs, I am afraid, just internal parts of the Julia kernel) seem to differ enough at times to make a binary built against, say, Julia 1.2 crash with 1.3 and vice versa.