Synchronize with Julia Yggdrasil

[mkitti]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:
Julia 1.6 currently uses MbedTLS version 2.24 and applies three patches.
https://github.com/JuliaPackaging/Yggdrasil/tree/master/M/MbedTLS

Julia Patches for 2.24.0:
https://github.com/JuliaPackaging/Yggdrasil/tree/master/M/MbedTLS/MbedTLS%402.24.0/bundled/patches

Julia 1.7 uses MbedTLS version 2.26 and applies a single patch:
https://github.com/JuliaPackaging/Yggdrasil/blob/master/M/MbedTLS/MbedTLS%402.24.0/bundled/patches/0001-Remove-flags-not-sopported-by-ranlib.patch

It would be great if this feedstock could provide a version of MbedTLS at the same versions and with similar build options and patches:
https://github.com/JuliaPackaging/Yggdrasil/blob/master/M/MbedTLS/common.jl

# enable MD4
sed "s|//#define MBEDTLS_MD4_C|#define MBEDTLS_MD4_C|" -i include/mbedtls/config.h
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=${prefix} \
    -DCMAKE_TOOLCHAIN_FILE="${CMAKE_TARGET_TOOLCHAIN}" \
    -DCMAKE_C_STANDARD=99 \
    -DUSE_SHARED_MBEDTLS_LIBRARY=On \
    -DENABLE_TESTING=OFF \

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

[izahn]

PRs more than welcome, or if you want to serve as a co-maintainer here I'm happy to add you. I need all the help I can get with this one!

[ngam]

@mkitti would you want this to be like a different output or should make this feedstock behave like julia's version? I guess we could output a MbedTLS-J or something...

@izahn opinion?

[izahn]

Separate branches for each version is the standard way to do it according to https://conda-forge.org/docs/maintainer/updating_pkgs.html#maintaining-several-versions. I prefer to follow those guidelines unless there is some compelling reason to do it differently.

[ngam]

Separate branches for each version is the standard way to do it according to https://conda-forge.org/docs/maintainer/updating_pkgs.html#maintaining-several-versions. I prefer to follow those guidelines unless there is some compelling reason to do it differently.

Understood re versions. Mainly concerned with the patches tho... Those may require a different output...

Or would you be okay with just adding them to all branches (versions) here?

[izahn]

Oh I see. Yeah, I'm OK with just applying the patches unconditionally, no need for separate outputs IMO.

[mkitti]

MbedTLS-J

If I were to call a Julia specific version something, maybe I would name it after https://github.com/JuliaBinaryWrappers/MbedTLS_jll.jl

The default Julia build process downloads that:
https://github.com/JuliaLang/julia/blob/master/deps/mbedtls.mk#L105

[izahn]

I'll leave it to your judgment. I've personally concluded that trying to get a julia ecosystem going in conda-forge is just not worth it given the julia community's hostility to third party builds. Happy to add you as a maintainer here, or merge any reasonable PRs.

[ngam]

I'll leave it to your judgment. I've personally concluded that trying to get a julia ecosystem going in conda-forge is just not worth it given the julia community's hostility to third party builds. Happy to add you as a maintainer here, or merge any reasonable PRs.

Totally understand @izahn... they're pretty hostile for some reason, see this from today for example: JuliaLang/julia#43666

But i think we are pretty close to a decent isolated julia ecosystem in conda. Is there something missing or still frustrating you?

[ngam]

Ironically... they sell themselves as "more open source" than python. Or like "truly" open source. LOL

[izahn]

My experience with third party julia builds is that they have problems and can't really be relied on. This isn't specific to conda-forge, I also have problems with the Archlinux build as well. For a brief moment I hoped we could get something solid in conda-forge, but for now I've decided to just manage my julia installs separately using the upstream binary releases.

[mkitti]

The Julia builds indicate very specific versions of all their dependencies, some of which are heavily patched. We have reached a point where enough of those patches have been upstreamed that it is becoming possible to do a third party build that kind of works. If they do not work, then better tests are needed. At the moment, conda-forge has omitted most of the stdlib tests for brevity.

In conda-forge, we have mostly ignored those version pins and patches for reasons that are beyond me. If we are going to get Julia to work in conda-forge, this attitude needs to change, on both sides.

[ngam]

If we are going to get Julia to work in conda-forge

We will, and we should definitely do ALL tests soon. I actually tried that and it was a mess. But that should be another focus area for you and me, @mkitti. More OS builds, and more rigorous testing. I think the whole binary-sharing, etc. stuff is a losing battle, but we will get to that later. I am going to add another issue for the testing in the feedstock (sorry izahn we invaded your feedstock 😆)

[izahn]

sorry izahn we invaded your feedstoc

No worries, happy to have you here!

[Keno]

Totally understand @izahn... they're pretty hostile for some reason, see this from today for example: JuliaLang/julia#43666

In principle, we have no problem with third party distributions. In fact, the Yggdrasil ecosystem was specifically designed to enable this. However, validating a Julia distribution is a significant effort - likely at least one full time person and a few $10k in hardware to make sure everything works. We do this for the official distribution, but third-party distributions do not in general have the resources to do this, so what ends up happening is that they ship broken distributions of julia. As such, our policy is that we do not offer support for builds other than the official ones and discourage third party distribution (other than re-distribution of the official binaries) for projects that do not have the resources to do proper validation or offer end user support.

[ngam]

@Keno understood. I hope it is clear to you that we are not trying to validate our distribution and have all our users go to you for help. This particular episode with openlibm is somewhat specific. Last time I checked, the founder of julia was running that openlibm repo, so the essential problem comes out of the same house.

[izahn]

In conda-forge, we have mostly ignored those version pins and patches for reasons that are beyond me.

Because conda-forge has it's own system of global pins designed to make packages within the conda-forge ecosystem compatible with each other. Are you planning to have julia-specific versions of all the dependencies, as you've proposed here, essentially creating a mirror Yggdrasil in conda-forge? That sounds like a lot of work for little gain, and at the end of the day your conda-forge julia package would probably not install alongside other popular conda-forge packages in the same environment.

[mkitti]

The objective is to narrow the differences between conda-forge and Yggdrasil where possible. In this specific case, we're within one small patch. I think we also need to move the Julia version forward to at least 2.28, and we may need 2.28 here as well.

[izahn]

The objective is to narrow the differences between conda-forge and Yggdrasil where possible.

I sincerely wish you luck with that. Basically julia has decided to package the world instead of working with third party packagers as most other open source projects do. I think that was a really bad idea, but that ship has sailed. The gulf now between the julia package ecosystem and others (conda-forge, linux distributions, etc.) is IMO too wide to be easily bridged. The task might be worth it if there was interest and collaboration from the julia side, but they've made it pretty clear that they have no interest in that.

In this specific case, we're within one small patch. I think we also need to move the Julia version forward to at least 2.28, and we may need 2.28 here as well.

I have no concerns about this specific case, go ahead and patch away! I raised the broader issues to explain why I won't be helping with this effort (other than to review and merge PRs in my capacity as maintainer of this feedstock).

[ngam]

fwiw, I personally agree with izahn on this though I would be happy to help as I am interested in experimenting and seeing where we can go. Plus as you may have seen in the linked issue on julia, I am outspoken and I will keep pinging them upstream to change their posture.

Last night, I was thinking of the following:

• we should continue pushing ahead with tighter integration, etc. and bridging the gap (this is the current plan)
• we should also produce a highly stable distribution of julia that replicates --- almost exactly --- their binaries. For that, we could simply pretend to be a user building without any system utilities, i.e. build all the extras as they want them. We can call that "julia-standalone" or "julia-copy" or something like that and have a strict condition that it won't run with the other julia we are working on.

The motivation: while yes, the tighter integration is nice to have, due to all sorts of problem from upstream, we ought to give the wide user base something stable and less experimental. Think of someone wanting to run some julia code on an HPC or personal machine. They don't care about all the packaging, etc., they will just something to work and work well.

@mkitti, pinning to exact deps like they do upstream won't be a viable solution for the julia with system utilities. It is jus too tasking and too restrictive, it will be borderline useless to go through the effort (i.e. just build what they want and don't rely on conda-provided solutions)

[ngam]

we should also produce a highly stable distribution of julia that replicates --- almost exactly --- their binaries. For that, we could simply pretend to be a user building without any system utilities, i.e. build all the extras as they want them. We can call that "julia-standalone" or "julia-copy" or something like that and have a strict condition that it won't run with the other julia we are working on.

For this, I will need to understand more things about how they build the extra tools and where. Obviously, there is a reason the linux distros and even brew.sh build against system utilities.

So basically we can have a julia formula, a la brew:

$ brew info julia
Warning: Treating julia as a formula. For the cask, use homebrew/cask/julia
julia: stable 1.7.1, HEAD
Fast, Dynamic Programming Language
https://julialang.org/
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/julia.rb
License: MIT and BSD-3-Clause and Apache-2.0 and BSL-1.0
==> Dependencies
Required: ca-certificates ✔, curl ✔, gcc ✔, gmp ✔, libgit2 ✘, libnghttp2 ✔, libssh2 ✔, llvm@12 ✘, mbedtls@2 ✘, mpfr ✔, openblas ✔, openlibm ✘, p7zip ✘, pcre2 ✔, suite-sparse ✔, utf8proc ✘
==> Requirements
Required: x86_64 architecture ✘
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 2,142 (30 days), 5,032 (90 days), 9,544 (365 days)
install-on-request: 2,149 (30 days), 5,042 (90 days), 9,551 (365 days)
build-error: 2 (30 days)

And a julia cask, a la brew (but this is just copying binaries fwiw, i.e. they get the .dmg file from official sources and simply copy/expand it to your mac)

$ brew info --cask julia
julia: 1.7.1
https://julialang.org/
Not installed
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/julia.rb
==> Name
Julia
==> Description
Programming language for technical computing
==> Artifacts
Julia-1.7.app (App)
/Applications/Julia-1.7.app/Contents/Resources/julia/bin/julia (Binary)
==> Analytics
install: 524 (30 days), 1,460 (90 days), 7,899 (365 days)

[mkitti]

pinning to exact deps like they do upstream won't be a viable solution for the julia with system utilities

What I expect is some concept of semantic versioning at least. I would not expect a dependency to be able to slip from 0.7 to 0.8. 1.7 to 1.8 might be acceptable.

[izahn]

• we should also produce a highly stable distribution of julia that replicates --- almost exactly --- their binaries.

It might be acceptable to just re-package the upstream binary directly. Some conda-forge feedstocks do that., e.g. https://github.com/conda-forge/firefox-feedstock

[ngam]

I sincerely wish you luck with that. Basically julia has decided to package the world instead of working with third party packagers as most other open source projects do. I think that was a really bad idea, but that ship has sailed. The gulf now between the julia package ecosystem and others (conda-forge, linux distributions, etc.) is IMO too wide to be easily bridged. The task might be worth it if there was interest and collaboration from the julia side, but they've made it pretty clear that they have no interest in that.

I suppose you agree with my comment here, right? JuliaLang/julia#43666 (comment). It basically makes it more or less impossible to use except if you get the binaries. I mean, it sure does sound like a protection of proprietary stuff type of maneuver, while retaining the "open-source" label.

I understand --- and maybe that's why you just vendor everything. But, I would hope that you start providing an actual range against which julia should work flawlessly. This is especially the case as there is a significant push to advertise julia --- this julia --- as a mature software. A mature software wouldn't be pinning deps to the sub-minor. This also goes against the spirit --- again as advertised by the appointed spokesperson of julia --- that julia is truly open source. What use is it if it is open source, yet has exacting and crazy deps and so you have to just get the binaries?

I am new to all of this; and I've been finding it somewhat confusing that some packages/software assume the world revolves around them. I found that in some random packages (e.g. vowpalwabbit and many Google open-sourced ones, as well as even multipass.run) where they basically bring in other packages as submodules (thus pointing to a commit) and this just makes it harder to figure out how to build it.

Anyway, I am commenting again because: Are you aware of guidelines against this sort of stuff? Or is open source truly the wild west? I know recently singularity joined the linux foundation and they had to make some changes --- but it turned out they are mostly logistic things about "community" stuff that I didn't really understand --- and so I wonder if leading open-source communities should publish guidelines against this stuff. Something like the fair-trade label.

Another related issue (imo) involves people not making their packages "noarch" when they truly are noarch --- thus ending up with unnecessary build costs and pins, as well as confusing matrix of exclusions (different OS archs, etc.). I've been making it a point to call packages out and convert them on conda-forge, but we may need a more systematic nudge / analysis tool to do that en masse.

Anyway, let me know if you have things I can refer to so that I can learn more about this, thanks!

[izahn]

We are well off-topic by now, but oh well.

I'm not so quick to suspect any bad motives on the part of the julia community. On the contrary, I think they are trying their best to provide a good experience for their users.

Mistrust of traditional open source packaging is by no means limited to julia, indeed the whole snap and flatpak ecosystems that red hat and ubuntu have been pushing stems from this same kind of mistrust that packagers will do a good job with your software. That mistrust has unfortunately been warranted sometimes, e.g., ubuntu does ship old and busted versions of some things. So developers get fed up and try to do the distribution themselves, using flatpak/snap/docker etc. to bundle all the dependencies. Julia is something of an outlier in having built its own software packaging and distribution system, but the spirit is part of a larger trend in open source. Some rants on this topic can be found at https://drewdevault.com/2019/12/09/Developers-shouldnt-distribute.html and https://ludocode.com/blog/flatpak-is-not-the-future . For the other side of the argument see e.g. https://github.com/Xpra-org/xpra/wiki/Distribution-Packages and https://mjg59.dreamwidth.org/41085.html

[mkitti]

If our make invocation were just make, that would be a supported configuration. This would download the precompiled binaries, and build Julia proper from source.

If you also wanted to build the dependencies from scratch, you could just do make USE_BINARYBUILDER=0. That would download dependency source, apply patches, and build each of them. Then Julia would be built from source. This is also a supported configuration.

You can disable binary builder for particular dependencies. make USE_BINARYBUILDER_CURL=0 would download a curl tarball and build it from scratch:
https://github.com/JuliaLang/julia/blob/4c45f292a0e688f7b32b1116f9feefc95f93f14d/deps/curl.mk#L15-L90
As you can see the tarball is downloaded directly from https://curl.haxx.se . The pre-built binary versions of the other dependencies would be downloaded.

Julia's build system specifies exactly where to retrieve the source and at what versions. That precise configuration is tested. That precise configuration is what is offered for download and is what is available in the source tarball.

What annoys me far more is when the build fails and I need to manually figure out the appropriate versions of dependencies that work or when there is no information about what versions of the dependencies have worked in the past at all. What is also confusing for me is that we are producing builds, and they are being combined with dependencies that we have not tested them with.

[izahn]

If our make invocation were just make, that would be a supported configuration. This would download the precompiled binaries, and build Julia proper from source.

Yes, I think that's right. But then what is the advantage over just grabbing the upstream binary?

[mkitti]

You can install it via conda or mamba?

[izahn]

Oh, I meant grabbing the upstream binary in the conda-forge package, i.e. just re-distributing the upstream binary as I suggested earlier.

[ngam]

You can install it via conda or mamba?

Yes, see izahn's comment above:

It might be acceptable to just re-package the upstream binary directly. Some conda-forge feedstocks do that., e.g. https://github.com/conda-forge/firefox-feedstock

https://github.com/conda-forge/firefox-feedstock/blob/9023becb51aca418f55237ef4bd524d67b4dd4fc/recipe/meta.yaml#L3-L17

Generally, you can do whatever you want, as long as you don't need sudo rights. That's my impression at least. And maybe you need some "relocatability" component.

Another example, see this build of blender and the discussion in there: conda-forge/staged-recipes#17162

[ngam]

Okay, the thing is. If we are going to do this, we have to do it right. First: julia is the official julia repackaging/rebuilding. Then we can just artificially fork our own julia and rename it and do the crazy enmeshing and all that.

[carlodri]

@ngam @mkitti I'm closing this due to long inactivity, if still relevant, feel free to reopen