Incompatibility between latest `julia=1.7.1` and latest `openlibm=0.8.0` on `conda-forge` #169

valeriupredoi · 2022-01-05T15:49:13Z

Hey guys, Happy New Year! Just a heads up that the latest Julia can not be installed in a conda env with deps from conda-forge unless one pins openlibm=0.7.0 (presumably a pin like openlibm<0.8 should work too) since the latest openlibm is now 0.8.0 as of 12 hours ago; see our issue on this matter ESMValGroup/ESMValTool#2450 Cheers 🍺

The text was updated successfully, but these errors were encountered:

ngam · 2022-01-05T17:37:43Z

Thanks. I was the one who pushed the 0.8.0 openlibm (and added arches there too). Should've paid attention. Should we do this pin more systematically somehow? I will make a quick fix for now.

isuruf · 2022-01-05T17:43:24Z

What's the actual error?

ngam · 2022-01-05T17:47:59Z

What's the actual error?

Looks like failed env solving: https://github.com/ESMValGroup/ESMValTool/runs/4715592816?check_suite_focus=true

ngam · 2022-01-05T17:49:44Z

  LinkError: post-link script failed for package conda-forge::julia-1.7.1-h989b2f6_1
1768
  location of failed script: /usr/share/miniconda3/envs/esmvaltool/bin/.julia-post-link.sh
1769
  ==> script messages <==
1770
  ERROR: Unable to load dependent library /usr/share/miniconda3/envs/esmvaltool/bin/../lib/julia/libopenlibm.so
1771
  Message:/usr/share/miniconda3/envs/esmvaltool/bin/../lib/julia/libopenlibm.so: cannot open shared object file: No such file or directory
1772

isuruf · 2022-01-05T17:51:14Z

Ah, yes, the ABI version changed in openlibm from 3 to 4.

ngam · 2022-01-05T17:52:52Z

Ah, yes, the ABI version changed in openlibm from 3 to 4.

Is this something we should pursue more systematically, a la libunwind? Or should the quickfix suffice for now?

ngam · 2022-01-05T18:26:09Z

Ah, yes, the ABI version changed in openlibm from 3 to 4.

Funny thing is... JuliaMath/openlibm#248 (comment)

ngam · 2022-01-05T18:26:38Z

CC @mkitti

ngam · 2022-01-06T04:37:14Z

@valeriupredoi this should be fixed. I will leave this open for now because we should pursue a more systematic solution with global pinning. However, it seems julia will drop openlibm pretty soon-ish.

valeriupredoi · 2022-01-06T12:31:43Z

@ngam brilliant, great many thanks! It indeed passes our tests no problemo. I noticed now openlibm=0.7.5 👍 🍺

valeriupredoi · 2022-01-06T12:33:18Z

cheeers muchly for your work, guys! Please close when you think it's ready to be closed, I am happy on our side of things 🍺

mkitti · 2022-01-06T12:34:14Z

I think @ngam is working on the global pinning now. I suppose we'll keep this open until that is complete.

valeriupredoi · 2022-01-06T12:40:44Z

whatever works for you guys, I'm a happy camper now that our envs get sorted 😁

ngam · 2022-01-06T14:32:22Z

@valeriupredoi I am still concerned about this. I suppose it passed your tests because it's making a new environment. In my older environment, it is still broken even if I update --all. I am not sure why it is not taking the latest julia build...

You should perhaps keep a strict pin on your end if you want to ensure your package works as expected for all users.

ngam · 2022-01-06T14:34:49Z

new env:

$ mamba create -n test julia

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.19.1) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['julia']

conda-forge/linux-64     Using cache
conda-forge/noarch       Using cache
Transaction

  Prefix: /home/ngam/.Mambaforge-Linux-x86_64/envs/test

  Updating specs:

   - julia


  Package                   Version  Build                Channel                    Size
───────────────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex               0.1  conda_forge          conda-forge/linux-64       3 KB
  + _openmp_mutex               4.5  1_gnu                conda-forge/linux-64      22 KB
  + arpack                    3.7.0  hdefa2d7_2           conda-forge/linux-64     Cached
  + bzip2                     1.0.8  h7f98852_4           conda-forge/linux-64     Cached
  + c-ares                   1.18.1  h7f98852_0           conda-forge/linux-64     Cached
  + ca-certificates       2021.10.8  ha878542_0           conda-forge/linux-64     Cached
  + curl                     7.80.0  h494985f_1           conda-forge/linux-64      90 KB
  + expat                     2.4.2  h9c3ff4c_0           conda-forge/linux-64     Cached
  + gettext                0.19.8.1  h73d1719_1008        conda-forge/linux-64     Cached
  + git                      2.34.1  pl5321hf874766_0     conda-forge/linux-64      13 MB
  + gmp                       6.2.1  h58526e2_0           conda-forge/linux-64     Cached
  + julia                     1.7.1  h989b2f6_2           conda-forge/linux-64     105 MB
  + krb5                     1.19.2  h48eae69_3           conda-forge/linux-64       1 MB
  + libblas                   3.9.0  12_linux64_openblas  conda-forge/linux-64      12 KB
  + libcblas                  3.9.0  12_linux64_openblas  conda-forge/linux-64      12 KB
  + libcurl                  7.80.0  h494985f_1           conda-forge/linux-64     339 KB
  + libedit            3.1.20191231  he28a2e2_2           conda-forge/linux-64     Cached
  + libev                      4.33  h516909a_1           conda-forge/linux-64     Cached
  + libffi                    3.4.2  h7f98852_5           conda-forge/linux-64     Cached
  + libgcc-ng                11.2.0  h1d223b6_11          conda-forge/linux-64     Cached
  + libgfortran-ng           11.2.0  h69a702a_11          conda-forge/linux-64      19 KB
  + libgfortran5             11.2.0  h5c6108e_11          conda-forge/linux-64     Cached
  + libgit2                   1.3.0  haabb1ae_1           conda-forge/linux-64     721 KB
  + libgomp                  11.2.0  h1d223b6_11          conda-forge/linux-64     Cached
  + libiconv                   1.16  h516909a_0           conda-forge/linux-64     Cached
  + liblapack                 3.9.0  12_linux64_openblas  conda-forge/linux-64      12 KB
  + libnghttp2               1.43.0  ha19adfc_1           conda-forge/linux-64     790 KB
  + libopenblas              0.3.18  pthreads_h8fe5266_0  conda-forge/linux-64      10 MB
  + libopenblas-ilp64        0.3.18  pthreads_h0aa5bea_0  conda-forge/linux-64     Cached
  + libssh2                  1.10.0  ha35d2d1_2           conda-forge/linux-64     233 KB
  + libstdcxx-ng             11.2.0  he4da1e4_11          conda-forge/linux-64     Cached
  + libunwind                 1.5.0  h9c3ff4c_1           conda-forge/linux-64     Cached
  + libutf8proc               2.7.0  h7f98852_0           conda-forge/linux-64     Cached
  + libzlib                  1.2.11  h36c2ea0_1013        conda-forge/linux-64     Cached
  + mbedtls                   3.1.0  h9c3ff4c_0           conda-forge/linux-64     Cached
  + metis                     5.1.0  h58526e2_1006        conda-forge/linux-64     Cached
  + mpfr                      4.1.0  h9202a9a_1           conda-forge/linux-64     Cached
  + ncurses                     6.2  h58526e2_4           conda-forge/linux-64     Cached
  + openblas-ilp64           0.3.18  pthreads_h3d04fff_0  conda-forge/linux-64     Cached
  + openlibm                  0.7.5  h7f98852_0           conda-forge/linux-64     208 KB
  + openssl                   3.0.0  h7f98852_2           conda-forge/linux-64       3 MB
  + p7zip                     16.02  he1b5a44_1000        conda-forge/linux-64     Cached
  + pcre2                     10.37  h032f7d1_0           conda-forge/linux-64     Cached
  + perl                     5.32.1  1_h7f98852_perl5     conda-forge/linux-64     Cached
  + suitesparse              5.10.1  h9e50725_1           conda-forge/linux-64     Cached
  + tbb                    2021.5.0  h4bd325d_0           conda-forge/linux-64     Cached
  + tk                       8.6.11  h27826a3_1           conda-forge/linux-64     Cached
  + zlib                     1.2.11  h36c2ea0_1013        conda-forge/linux-64     Cached

  Summary:

  Install: 48 packages

  Total download: 135 MB

───────────────────────────────────────────────────────────────────────────────────────────

Confirm changes: [Y/n] n
Aborted.

ngam · 2022-01-06T14:36:59Z

old env even after update --all multiple times:

julia                     1.7.1                h989b2f6_1    conda-forge
openlibm                  0.8.0                h7f98852_0    conda-forge

It basically taking this as more of a stable position for the current env.

ngam · 2022-01-06T14:37:39Z

This is why we urgently need global pinning, I suppose...

ngam · 2022-01-06T14:40:20Z

@wolfv @isuruf @jakirkham quick question about how mamba/conda resolves update --all calls vis-a-vis global pinning:

We had an issue yesterday where one dep (openlibm) slyly changed its abi number when it moved to 0.8.0 and thus it ended up breaking julia. We pinned openlibm in julia (here) to avoid that bad openlibm as an interim solution while we work on global pinning for openlibm. In a new env (see above), this works fine, but with old env (via update --all call) this doesn't seem to fix the issue. I suppose this calls for global pinning asap? Or is this an underlying bug that should be fixed elsewhere?

cc @conda-forge/core

jaimergp · 2022-01-06T14:51:46Z

Can you target the desired version directly? conda install julia openlibm<0.8? The older Julia builds might need a repodata patch too, because right now they are less constrained and conda likes maximizing the versions, so the solver might be choosing that (more recent version for openlibm wins over more recent build for julia?). It also minimises the number of changes, by the way, so one more reason to choose the "simpler" solution, even if it's not technically correct anymore.

TLDR: repodata patch might be needed.

mkitti · 2022-01-06T14:56:13Z

Is it possible to just pull openlibm 0.8 for the moment? Is there anything that is actually using it?

jaimergp · 2022-01-06T14:58:07Z

Can't really say, but with the CDN sync delays it will take a bit to show up for users. It might be as fast to just patch Julia packages here.

beckermr · 2022-01-06T15:13:15Z

This needs a repodata patch. We should avoid pulling deps and builds of big packages like these.

ngam · 2022-01-06T15:19:22Z

Can you target the desired version directly? conda install julia openlibm<0.8? The older Julia builds might need a repodata patch too, because right now they are less constrained and conda likes maximizing the versions, so the solver might be choosing that (more recent version for openlibm wins over more recent build for julia?). It also minimises the number of changes, by the way, so one more reason to choose the "simpler" solution, even if it's not technically correct anymore.

TLDR: repodata patch might be needed.

Makes sense, thanks!

Yes, the direct solution works.

ngam · 2022-01-06T15:32:44Z

This needs a repodata patch. We should avoid pulling deps and builds of big packages like these.

Thanks @beckermr, see pr:
conda-forge/conda-forge-repodata-patches-feedstock#208

valeriupredoi · 2022-01-06T15:41:20Z

as @jaimergp points out conda (mamba) likes to go up in version numbers - that's why when you do an update it will try and maximize the deps' versions, as long as there are no conflicts and the env gets solved: note that in that case julia stays at the slightly older build h989b2f6_1 but openlibm goes to 0.8.0. AFAIK update --all is coming packed with less checks than a full blown env build and solve, that's why updating an env is not what we recommend to users since it may result in an otherwise fit environment to break after the update. At our end, as users of the julia conda package, we should be fine since we always ask the users to create envs from scratch, and the CI tests we do either use pre-built containers (with release point environments, which are inherently old, but stable) or the test suites create new envs every night. It may be a problem for actual active Julia users that decide to do an env update when they bring in some third party package they need in their environment, that's why I was trying to tell that Julia developer about this incompatibility in the issue I opened yesterday in the main Julia GH repo, but they said it's all fine and closed it 🤷‍♂️

isuruf · 2022-01-06T15:42:32Z

Given that JuliaMath/openlibm#248 says that the SONAME bump was a mistake, we can just make a symlink

ngam · 2022-01-06T15:45:58Z

Given that JuliaMath/openlibm#248 says that the SONAME bump was a mistake, we can just make a symlink

Definitely, but... like they did and decided it was the lesser of two evils to just go ahead with the mistake, we might as well follow their suit, otherwise, we will forget to disable the symlink, etc.

valeriupredoi · 2022-01-07T13:33:50Z

hey guys, heads up that now Julia installation from conda-forge (the updated _2 build) results in a segfault - see the GA test I just ran, look under third step/Updating 'esmvaltool' env from conda env update... even though openlibm is now at the same version as yesterday (openlibm 0.7.5 h7f98852_0 conda-forge/linux-64 208 KB) - must be something else that changed overnight? Cheers 🍺

mkitti · 2022-01-07T13:55:57Z

@SylvainCorlay just reported the same thing
#159 (comment)

The errors all point to a segfault in curl, and curl 7.81 was released.

ngam · 2022-01-07T15:04:59Z

https://github.com/JuliaLang/julia/blob/4f1ff0bb0055806fcce880765c0be26313f936bf/deps/Versions.make#L28-L30

upstream is 7.73.0

ngam · 2022-01-07T15:07:00Z

Okay, gonna add curl <7.81 to repo patch now: conda-forge/conda-forge-repodata-patches-feedstock#208

ngam · 2022-01-07T15:28:52Z

More reports of same issue, from archlinux: JuliaLang/julia#43682

valeriupredoi · 2022-01-07T16:33:09Z

cheers muchly @ngam - apologies I didn't pick up on the version upping of curl myself so I can report it to you, got distracted with other stuffs. I am a bit bewildered by the approach from core Julia people brushing all this under the carpet by saying the conda package is not official distribution, but heyho, not my place to comment, you guys are doing a great job! 🍺

ngam · 2022-01-07T16:51:51Z

I am a bit bewildered by the approach from core Julia people

LOL it's okay. They've been somewhat unfriendly (we discussed this a little bit here: conda-forge/mbedtls-feedstock#1), but we will figure things out soon. We are (in conda-forge/julia-feedstock) serious about having this work flawlessly. For now, we are playing catchup with issues emerging (libunwind, openlibm, curl, who know what's next tomorrow!), but hopefully soon we will figure out a more systematic approach.

The ultimate problem isn't the people upstream per se, it is more their design choices of how julia is built and shipped. They ship julia with exact pins (and vendoring everything). I am quite outspoken and I don't mind speaking up repeatedly about this. This is really a losing strategy for julia and the open-source community at large: We all want julia to succeed and become a model for open-source collaboration; we don't want julia to become the next Matlab

ngam · 2022-01-07T16:58:53Z

cheers muchly @ngam - apologies I didn't pick up on the version upping of curl myself so I can report it to you, got distracted with other stuffs.

Hey, you've been extremely helpful. Please keep them coming and as @mkitti mentioned, we are going to be proactive about ensuring your work is tested by us before updating/merging.

valeriupredoi · 2022-01-07T16:59:35Z

the next Matlab

🤣

Keep up the good work, I'll try help as much as I can, even though that might just be poking you and complaining things are broken again 😁

ngam · 2022-01-09T02:23:33Z

Status update:

We are waiting for update openlibm dep for julia versions conda-forge-repodata-patches-feedstock#208 to be merged for openlibm
We have effectively fixed the curl issue (PR fixing the curl issues #178 and fix curl issues into v1.6 #179), except it is going to be quite hard to go back and fix this for older versions. If one cannot upgrade, a quick fix is to use a different "Downloads.jl" module following these instructions: https://github.com/JuliaLang/Pkg.jl/blob/7af3e27e3784ccde287b4bcb87891a22830a7dee/README.md#using-the-development-version-of-pkgjl (@mkitti can help further)

valeriupredoi · 2022-01-10T11:57:01Z

Morning guys, heads up that your latest fix in the _3 build works nicely and we have green CI tests again - with curl=7.81 and _3 Julia build, see the env - many thanks for the quick response! 🍺

See: * JuliaLang/Downloads.jl#172 * conda-forge/julia-feedstock#169 (comment) PR: 261417 Reported by: rizor.reardon@yahoo.com

mkitti · 2022-02-09T01:55:49Z

@valeriupredoi and @ngam , I believe this issue can be closed since the above pull requests have been merged. Feel free to reopen if that is not the case.

ngam mentioned this issue Jan 5, 2022

pin openlibm to under 0.8.0 #170

Merged

5 tasks

This was referenced Jan 5, 2022

pin openlibm to below 0.8.0 #171

Closed

[Julia] Julia installation from conda-forge throws SegFault and unsucessful conda rollback ESMValGroup/ESMValTool#2450

Closed

ngam mentioned this issue Jan 6, 2022

update openlibm dep for julia versions conda-forge/conda-forge-repodata-patches-feedstock#208

Merged

5 tasks

jaimergp mentioned this issue Jan 7, 2022

curl v7.81.0 conda-forge/curl-feedstock#101

Merged

3 tasks

ngam mentioned this issue Jan 7, 2022

proposing a version bump of libcurl to 7.81 JuliaLang/julia#43703

Closed

freebsd-git pushed a commit to freebsd/freebsd-ports that referenced this issue Jan 25, 2022

lang/julia: unbreak build when using libcurl 7.81

175c1f5

See: * JuliaLang/Downloads.jl#172 * conda-forge/julia-feedstock#169 (comment) PR: 261417 Reported by: rizor.reardon@yahoo.com

5u623l20 pushed a commit to 5u623l20/freebsd-ports that referenced this issue Jan 27, 2022

lang/julia: unbreak build when using libcurl 7.81

90d2595

See: * JuliaLang/Downloads.jl#172 * conda-forge/julia-feedstock#169 (comment) PR: 261417 Reported by: rizor.reardon@yahoo.com

MilesCranmer mentioned this issue Feb 1, 2022

Add 0.7.0; fix openlibm requirement conda-forge/pysr-feedstock#2

Closed

5 tasks

mkitti closed this as completed Feb 9, 2022

Incompatibility between latest julia=1.7.1 and latest openlibm=0.8.0 on conda-forge #169

Incompatibility between latest julia=1.7.1 and latest openlibm=0.8.0 on conda-forge #169

Comments

valeriupredoi commented Jan 5, 2022

ngam commented Jan 5, 2022

isuruf commented Jan 5, 2022

ngam commented Jan 5, 2022

ngam commented Jan 5, 2022

isuruf commented Jan 5, 2022

ngam commented Jan 5, 2022

ngam commented Jan 5, 2022

ngam commented Jan 5, 2022

ngam commented Jan 6, 2022

valeriupredoi commented Jan 6, 2022

valeriupredoi commented Jan 6, 2022

mkitti commented Jan 6, 2022

valeriupredoi commented Jan 6, 2022

ngam commented Jan 6, 2022 • edited

ngam commented Jan 6, 2022

ngam commented Jan 6, 2022

ngam commented Jan 6, 2022

ngam commented Jan 6, 2022 • edited

jaimergp commented Jan 6, 2022 • edited

mkitti commented Jan 6, 2022

jaimergp commented Jan 6, 2022

beckermr commented Jan 6, 2022

ngam commented Jan 6, 2022

ngam commented Jan 6, 2022 • edited

valeriupredoi commented Jan 6, 2022

isuruf commented Jan 6, 2022

ngam commented Jan 6, 2022

valeriupredoi commented Jan 7, 2022

mkitti commented Jan 7, 2022

ngam commented Jan 7, 2022 • edited

ngam commented Jan 7, 2022 • edited

ngam commented Jan 7, 2022

valeriupredoi commented Jan 7, 2022 • edited

ngam commented Jan 7, 2022 • edited

ngam commented Jan 7, 2022

valeriupredoi commented Jan 7, 2022

ngam commented Jan 9, 2022 • edited by mkitti

valeriupredoi commented Jan 10, 2022

mkitti commented Feb 9, 2022

Incompatibility between latest `julia=1.7.1` and latest `openlibm=0.8.0` on `conda-forge` #169

Incompatibility between latest `julia=1.7.1` and latest `openlibm=0.8.0` on `conda-forge` #169

ngam commented Jan 6, 2022 •

edited

ngam commented Jan 6, 2022 •

edited

jaimergp commented Jan 6, 2022 •

edited

ngam commented Jan 6, 2022 •

edited

ngam commented Jan 7, 2022 •

edited

ngam commented Jan 7, 2022 •

edited

valeriupredoi commented Jan 7, 2022 •

edited

ngam commented Jan 7, 2022 •

edited

ngam commented Jan 9, 2022 •

edited by mkitti