Skip to content

Updating JLLs with Dependency("libcxxwrap_julia_jll") #2160

New issue
New issue
Closed
Closed
Updating JLLs with Dependency("libcxxwrap_julia_jll")#2160
@fingolfin

Description

@fingolfin
fingolfin
opened on Nov 21, 2020

Once we have versions of libcxxwrap_julia_jll for all Julia versions, we'll need to update the various JLLs using it to also provide multiple versions. Here I try to describe what's going on and why a change is needed; explain how to make that change; and track the progress of each affected package.

What's going on

Julia does not currently provide a stable ABI, meaning that things that were compiled against, say, Julia 1.3, may or may not work when linked at runtime against Julia 1.4/1.5/... -- and this holds of course for any combination of Julia versions. For libcxxwrap and anything using its headers (including all (?) JLLs using Dependency("libcxxwrap_julia_jll")), it is indeed a fact that a binary compiled against Julia 1.3 can lead to segfaults during GC when used with any later Julia version, and vice versa. Right now it is not definitely known whether other Julia combinations can lead to issues, but it is not unlikely that this will eventually happen.

Thus the only safe way forward is to provide binaries for libcxxwrap_julia_jll which are compiled against each of Julia 1.3, 1.4, 1.5, ....

In an ideal world, the julia version would simply be part of the Platform, and we'd be done. Indeed, @staticfloat added support for that -- alas, it will only work in Julia >= 1.6; older versions will balk when they see such platform identifiers. So as long as one wants to support any older Julia versions (which I'd expect to be everybody right now, as 1.6 has not yet been released, not even branched), extra work is needed.

On the upside to all this, cxxwrap now supports many more platforms, which you can potentially benefit from for your own JLLs, too!

How to deal with it

The following is an initial suggestion, and will be revised based on feedback; I suggest that people wait a bit with trying to implement this, until we've ironed out everything

In a nutshell, here's the plan: for each affected JLL, we build it several times, once for each supported Julia minor version (so e.g. one each for Julia 1.3, 1.4 and 1.5+), but built from the same source. For technical reasons, each such build will have to carry a separate version. The convention I propose is to set the patch level of your JLL to the minor version of Julia (so libcxxwrap_julia_jll 0.8.4 was built against Julia 1.4). Yes, that means you can't have a one-to-one mapping with the actual versions of the code your JLL wraps; but the longterm plan for JLLs is to completely decouple JLL versions from the versions of the wrapped software anyway.

There are many ways to deal with this; I'll describe one of them here, which I am using in PR #1798. The core idea is to rename your current build_tarballs.jl to common.jl and then add one subdir for each supported Julia version, each with a build_tarballs.jl including common.jl.

For example, PR #1798 renames L/libsingular_julia/build_tarballs.jl to L/libsingular_julia/common.jl and adds L/libsingular_julia/libsingular_julia@1.3/build_tarballs.jl which looks like this:

const julia_version = v"1.3.1"
include("../common.jl")

For technical reasons, the build_tarballs.jl for Julia 1.4 and 1.5 will be added in later PRs (AFAIK Yggdrasil tooling does not allow adding multiple versions of a JLL in a single PR -- so far that would have been a weird thing to do anyway).

In common.jl, some changes are made:

  • the version is adjusted to set the patchlevel to match the Julia minor version: version = VersionNumber(0, 2, julia_version.minor)
  • the list of platforms is expanded to match the platforms supported by libcxxwrap_julia_jll (this could be neater, see also Add a way to "inherit" default platform list from another JLL BinaryBuilder.jl#957)
  • the dependencies are adjusted:
    • instead of Julia_jll, we now need libjulia_jll in the right version:
      BuildDependency(PackageSpec(name="libjulia_jll", version=julia_version)),
  • we inform build_tarballs about the runtime Julia requirements, by adding this keyword argument:
    julia_compat = "$(julia_version.major).$(julia_version.minor)"
    Note that this is not using ~ as one might expect, simply so that the build for Julia 1.5 can be used with Julia 1.6 until a libjulia_jll for 1.6 exists (yes, that might not work, due to ABI incompatibilities; but at least it has a chance of working, which is better than what we'd have if we used ~ in the version specification. Technically this means one could also use the JLL version made for Julia 1.3 in Julia 1.4; but since the JLL version for Julia 1.4 is strictly newer than that for Julia 1.3, this shouldn't happen unless a user forces that, and well, then they deserve it...

UPDATE: we also recommend that all packages using libcxxwrap_julia_jl should use preferred_gcc_version=v"8" or higher; see #2236 and JuliaInterop/libcxxwrap-julia#75

Affected JLLs and their status

Note: I plan to ping the respective authors once we have a stable plan, but for now, it seems premature to bother them. In particular until PR #2158 is ready...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions