[julia]: Update build script to be compatible with new BB

[jstrube]

Not sure about the placement of preferred_gcc_version. It's not documented, so I guessed after looking at the source.

[ahumenberger]

Why not platforms = supported_platforms()?

[ViralBShah]

We don't build Julia for all the supported platforms upstream (such as musl).

[ahumenberger]

The problem seems to be that Julia cannot dlopen another version of libjulia in https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/a8132866f9072508265d79f98d735885dd75b5b7/src/Auditor.jl#L114

[staticfloat]

Unfortunately, Julia cannot be cross-compiled, because the bootstrap portion of the build doesn't support cross-compilation. It is possible to do it with WINE, but we don't have that setup in BB yet. :(

[ViralBShah]

Can we make BB serve the regular Julia binaries (without trying to compile them)?

[Gnimuc]

Yes, I think we can do what HDF5_jll did here as a temporary workaround.

[jstrube]

I like the idea of just serving the regular Julia binaries. I understand how to download the tarballs for Linux and FreeBSD and extract them into the right dir.
How do I deal with the Windows executable and the MacOS dmg?

[giordano]

I think 7zip should be able to unpack a dmg

[ViralBShah]

I believe the right thing to do here is to make the Julia buildbots upload tarballs for all platforms.

In the meanwhile I do believe you can find Linux tools to extract files from Windows installer executables. Maybe 7zip.

[jstrube]

One important aspect of this is that it would have to package the includes. Am I right, @barche?
Not sure how a jll would accomplish that... Thoughts?

[barche]

@ViralBShah I second the idea of using the official binaries, but extracting the official binaries on Linux failed for me for mac and/or Windows when I tried it (around the release of Julia 1.0, I think I tried with 7z).

@jstrube Yes, the includes are needed, but they are there in the official distribution and should also be in the jll tarball?

[ViralBShah]

I wonder what the CI systems do on mac and windows to get the official Julia binaries. Perhaps we can do something similar.

[giordano]

I wonder what the CI systems do on mac and windows to get the official Julia binaries.

They have native systems, we don't

[ViralBShah]

I wonder what the CI systems do on mac and windows to get the official Julia binaries.

They have native systems, we don't

I thought there may be some intermediate tarballs from the buildbots that they might be using. But that is not the case - https://github.com/JuliaCI/install-julia/blob/master/install-julia.sh

[staticfloat]

It looks like our .dmg files are very unfriendly to extraction with 7z. You can apk add p7zip to get it in the build environment.

• To extract from .exe files, you do 7z x <path to .exe>. That will extract another .exe, which you do 7z x <path to new .exe> on again. That will result in the bin, etc, include, etc... directories plopped into the current directory.

• To extract from .dmg files, you can do the same, but one of the hfs filesystems it tries to extract is HUGE uncompressed (I stopped it after it tried to extract 50GB of zeros) so we would need to use a different tool here, I think.

Viral is probably right in that the best thing for us to do is to generate tarballs on the buildbots.

[jstrube]

I don't understand where this build error comes from: "can't dlopen library" https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=1283&view=logs&j=ed5bc0cf-9e11-56fd-407f-0f551f71f6db&t=afded763-3b68-5ccd-6b86-d980edfd7e43&l=1893
Especially on x86_64-linux-gnu this shouldn't happen... At least that one worked for me locally...

[ViralBShah]

Ok, who can help with making this happen?

[barche]

Ok, who can help with making this happen?

The problem is not that it's hard or much work, but to repackage the official Julia release as a JLL, we need .tar.gz files for all platforms, and MacOS and Windows are missing (available as dmg and .exe instead). So it needs intervention from someone who can make the official binary build system generate these. Ideally, they should also use a naming scheme compatible with the $target variable in BB, but that's optional since with judicious use of the shell case statement it's easy enough to work around.

[ViralBShah]

@staticfloat Do you know who in the JuliaCI / buildbot area can help upload tarballs of these binaries going forward? For now, should we just do it manually?

[staticfloat]

We've made some progress on master recently, @musm has been helping to unify the make binary-dist targets; we should see if we can get the buildbots to upload not only the .exe and .dmg forms of Julia, but also the .tar.gz. @musm sound like something fun to do? I'd be happy to walk you through the necessary modifications to https://github.com/JuliaCI/julia-buildbot/blob/master/master/package.py

[ViralBShah]

Perhaps @DilumAluthge can help?

[giordano]

Superseded by #1048

[staticfloat]

https://github.com/JuliaBinaryWrappers/Julia_jll.jl

[ViralBShah]

It would perhaps be better to start using these in CI and such. @DilumAluthge

[ViralBShah]

@staticfloat Do these get uploaded automatically when we build a new release? Not seeing 1.4.2.

@ararslan

[staticfloat]

No, someone has to update the builder in Yggdrasil and push a build through.

[ViralBShah]

I meant the upstream ArchiveSources - those are the ones that are failing.

[ViralBShah]

Opened #1084 to figure out how to get Julia 1.4.2 in.