[julia] julia binaries for BB packages that need them as dependencies

[jstrube]

As far as I understand, Cxx and CxxWrap need access to julia binaries at build time.
There is a build recipe here in Yggdrasil, but that doesn't work, because julia can't be cross-compiled.
There's also a repo to host binaries here: https://github.com/JuliaPackaging/JuliaBuilder, but that hasn't been updated with new binaries in a while, so @Gnimuc forked it and put some newer binaries there https://github.com/Gnimuc/JuliaBuilder/releases.
The new BB can't use the build recipes as binary dependencies any more, so after a discussion on slack, I tried to create an Artifacts.toml by hand, PR is here: JuliaRegistries/General#7030. It was closed without a suggestion for a better place to put the binaries.
All of that is a bit of a mess.
I think it would be good to have one canonical location for julia binaries so that they can be pulled in as binary dependencies by BB scripts. I've tried to collect the references to all of the relevant locations in this issue, hopefully that helps with retiring some of them.
cc @barche @giordano @staticfloat

[staticfloat]

I think we probably want to create something similar to CompilerSupportLibraries, where we have a JLL that contains things like the headers and libraries for Julia.

While in the case of CSL, it's mostly okay that depending on it downloads a second copy of e.g. libgcc_s, it's much less desirable here because libjulia is big, and it would be wasteful to spend time and effort downloading it.

@giordano I'm thinking perhaps we should add a way to specify that a dependency should not be tracked in the JLL package. What do you think?

[barche]

@giordano I'm thinking perhaps we should add a way to specify that a dependency should not be tracked in the JLL package. What do you think?

Maybe build-time vs runtime dependencies, much like Arch PKGBUILD files have?

[vchuravy]

Buildtime dependencies would be much appreciated, things like capnproto are both like a library and a compiler that you use during building your binaries.

…

On Sun, Dec 22, 2019, 11:39 Bart Janssens ***@***.***> wrote: @giordano <https://github.com/giordano> I'm thinking perhaps we should add a way to specify that a dependency should not be tracked in the JLL package. What do you think? Maybe build-time vs runtime dependencies, much like Arch PKGBUILD files have? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#321?email_source=notifications&email_token=AABDO2TYLIV67NZMPG4UH23QZ47U5A5CNFSM4J6JVRUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHPNA7Y#issuecomment-568250495>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDO2XH6U6WFTPYVT24SBTQZ47U5ANCNFSM4J6JVRUA> .

[giordano]

Yes, build-time dependencies are a frequent need.

In #110 I initially did an (almost) native build for x86_64-linux-gnu separately, put in a tarball, and used that as additional source. In the end I managed to not use an external source.

Having a native build already available for bootstrap would also avoid the troubles we had in #319 when targeting macOS. I think Valentin is having similar problems for another package

[ViralBShah]

@ChrisRackauckas was also asking for this.

[giordano]

The more I think about compile-time dependencies, the more I'm convinced that in most cases they can be regarded more as an additional source than as a regular dependency, i.e. like something that doesn't necessarily need to live in ${prefix}: these compile-time dependencies are often utilities/programs that can be referenced with the absolute path and don't need to stay in ${bindir}.

However there would probably be cases where they need to be treated as regular dependencies installed in ${prefix}.

[ViralBShah]

I was hoping that this would enable us to be able to download Julia through the BB infrastructure for running stuff as well (not just compile-time). Like in JuliaCall (for R) or pyJulia.

[giordano]

I was hoping that this would enable us to be able to download Julia through the BB infrastructure for running stuff as well (not just compile-time). Like in JuliaCall (for R) or pyJulia.

Yes, the build-time dependencies is an orthogonal, but somewhat connected, issue to this one.

[ViralBShah]

I am thinking that in our Julia buildbots, before we build the installers, we create a directory called julia-<commit>, which could be tarballed and uploaded for this purpose. From there on, the rest is perhaps straightforward. Coming to think of it, much of this probably already exists for the CI infrastructure.

For full reproducibility then, one can use any download method to get the relevant Julia tarball, and then give the untarred Julia a Manifest file.

[staticfloat]

I think build-time dependencies fall into two camps:

The first is programs/utilities that need to run in order to generate stuff during the build. Things like make, gcc, tblgen (for LLVM), etc... These I think Mosé is correct on in saying that we can treat them like "extra sources". We already have people doing this and it is mostly working.

The second camp is dependencies that are (or will be) automatically shipped within Julia (e.g. Julia_jll ,OpenBLAS_jll, MbedTLS_jll, CompilerSupportLibraries_jll, etc....). Things that are already present, so we don't actually need to download or load them. For this, I can see multiple ways of dealing with it (have Julia ship with some artifacts/JLLs implicitly present in the stdlib depot, have Pkg ignore certain blacklisted JLLs, have a way for BinaryBuilder to not emit a dependency in a JLL implicitly trusting that it will exist on this Julia version, etc....).

[ViralBShah]

Should we execute the first step of getting tarballs of Julia binaries on Mac and Windows while we figure out the rest? That seems like it should be pretty straightforward - but for the people who are familiar with the build systems. If someone can give me directions on how to do it, I'm happy to give it a shot as well.

[fingolfin]

I really also could use this (see also JuliaPackaging/BinaryBuilder.jl#511)