libstdcxx-ng versioning

[kllrak]

Thanks for the excellent package! I recently ran into a situation where I tried to install a Python package using CondaPkg.jl that depended upon libstdcxx-ng v12 and was unable to install it due to incompatible dependencies. This resulted in some error about GLIBCXX_3.4.30 not being found. I think I could in principle resolve this by building Julia from source with a more recent version of libstdc++, but because of the lines below I don't think it would work even then. Would it be a reasonable idea to actually run the strings command and find the highest GLIBCXX version string in PythonCall?

https://github.com/cjdoris/PythonCall.jl/blob/77ee463d149b418ed1e6b34036f50172429ae46e/src/cpython/context.jl#L67-L80

Also, the link in the comment on L70 doesn't contain GLIBCXX_3.4.30, however I do see it mentioned here: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

[cjdoris]

Indeed those lines you found assume you are using a standard build of Julia. As I think you worked out, Julia ships it's own libstdc++, so we need to make sure any Python packages don't rely on newer versions, and therefore PythonCall will add a Conda dependency on a suitably old version of libstdc++.

PythonCall can't do anything else about it because libstdc++ is already loaded by the time PythonCall is loaded.

You could ask the other package to build against an older version of libstdc++. It's unlikely they really need the latest version.

Other than that, I think your only option is to build Julia yourself with a newer libstdc++. I'd be happy to add an override for the extra Conda dependency that PythonCall makes.

[hhaensel]

I found that the best approach is not to restrict oneself to the old libstdc++ library but to exchange julia's version by a new one.

This little script does all necessary steps

using PythonCall

run(PythonCall.C.CondaPkg.conda_cmd(`install -c conda-forge -y libstdcxx-ng`))

conda_lib = joinpath(dirname(PythonCall.C.CTX.lib_path), "libstdc++.so.6")
julia_lib = joinpath(dirname(Sys.BINDIR), "lib", "julia", "libstdc++.so.6")

rm(julia_lib)
symlink(conda_lib, julia_lib)

Note that libcstdcxx-ng is installed via a run command and not via the CondaPkg, because PythonCall currently enforces a lower version as another workaround to this problem.

After having run this script exit Julia and restart. Now everything should work fine. My test line is

using PythonCall
pyimport("pyarrow")

Note that after any new installation of Python packages you have to rerun

run(PythonCall.C.CondaPkg.conda_cmd(`install -c conda-forge -y libstdcxx-ng`))

[hhaensel]

This page helped me a lot.

Meanwhile there's a PR in place to have a check running during julia startup.

[hhaensel]

I published a Package that helps replacing the symlinks of libstdc++.so:

ReplaceLibstdcxx.jl

It is scheduled for registration after 3 days from now ...
Don't hesitate to comment if something is missing or failing.

I think one could release the restriction of supplying the libstdcxx-ng<11.4 and rather force libstdcxx-ng>=11.4 and integrate replace_libstdcxx(PythonCall) from my package.

[hhaensel]

I learned that there is a PR that will make this package obsolete. I will therefore not register the package with the General Registry.

[cjdoris]

IIUC, those PRs will (a) bump the version of libstdc++ bundled with Julia; and (b) use your system libstdc++ if it is newer than the one bundled with Julia. Is that right?

If so, I can increase the bounds in PythonCall when the new version of Julia is out. Lifting the bound further dynamically if (b) happens will be harder though. Perhaps an env var to control the Conda version of libstdc++ will suffice?

[cjdoris]

#243 should fix this.