Binary compatibility between conda-forge and defaults

[jjhelmus]

The topic of binary compatibility between the packages in the conda-forge and defaults has been often discussed in feedstocks, mailing lists and in-person but I do not believe there is an issue for discussing the issue in general. I wanted to create such an issue to document these incompatibilities, collect some representative examples and discuss/track progress on a solution.

---

Currently, the linux-64 packages in conda-forge are compiled using the GCC 4.8.2 toolchain from devtoolset-2 inside a CentOS 6 docker container.

The packages in defaults are compiled using a GCC 7.2 toolchain created using crosstools-ng and distributed as conda packages. Builds are done inside a CentOS 6 docker container.

These two toolchains use different C++ ABIs. The GCC 7.2 toolchain used by defaults uses the "new" ABI where as the devtoolset-2 toolchain in conda-forge uses the "old" ABI. Passing the -D_GLIBCXX_USE_CXX11_ABI=0 argument can be used to change which ABI is produced by the GCC 7.2 compiler but -D_GLIBCXX_USE_CXX11_ABI=1 cannot be used to change the ABI target of the devtoolset-2 toolchain. These two ABI are not compatible and mixing libraries and binaries with different ABIs will fail. Specifically, the linker will not be able to resolve symbols as different symbol names are used by the two ABIs. Compute Canada has a nice document providing more details on the dual C++ ABI.

mosh provides a nice example of this incompatibility as it links to libprotobuf, a C++ library. The mosh-client binary works when used with the libprotobuf package from the conda-forge channel which uses the old C++ ABI:

~$ conda create -y -q -n example -c conda-forge mosh
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/jhelmus/miniconda3/envs/example

  added / updated specs: 
    - mosh


The following NEW packages will be INSTALLED:

    ca-certificates: 2018.4.16-0           conda-forge
    libgcc-ng:       7.2.0-hdf63c60_3      defaults   
    libprotobuf:     3.5.2-hd28b015_1      conda-forge
    libstdcxx-ng:    7.2.0-hdf63c60_3      defaults   
    mosh:            1.3.2-pl526h5cde1c9_1 conda-forge
    ncurses:         6.1-hfc679d8_1        conda-forge
    openssl:         1.0.2o-h470a237_1     conda-forge
    perl:            5.26.2-h470a237_0     conda-forge
    zlib:            1.2.11-h470a237_3     conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
~$ conda activate example
(example) ~$ mosh-client 
mosh-client (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Usage: mosh-client [-# 'ARGS'] IP PORT
       mosh-client -c

The binary fails with the libprotobuf package from defaults which uses the new C++ ABI.

(example) ~$ conda install -y -q defaults::libprotobuf
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/jhelmus/miniconda3/envs/example

  added / updated specs: 
    - defaults::libprotobuf


The following packages will be DOWNGRADED:

    ca-certificates: 2018.4.16-0       conda-forge --> 2018.03.07-0      defaults
    libprotobuf:     3.5.2-hd28b015_1  conda-forge --> 3.5.2-h6f1eeef_0  defaults
    openssl:         1.0.2o-h470a237_1 conda-forge --> 1.0.2o-h20670df_0 defaults

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
(example) ~$ mosh-client 
mosh-client: symbol lookup error: mosh-client: undefined symbol: _ZNK6google8protobuf11MessageLite25InitializationErrorStringEv

A few C++ libraries that I am aware of that are incompatible between conda-forge and defaults are:

• libprotobuf/protobuf
• boost
• gflags
• glog

---

I do not believe the different toolchains produce incompatible C libraries or binaries.

I am not aware of any Fortran incompatibilities but would not be surprised if they exist as the ABI was broken in GCC 7.

[ihnorton]

The packages in defaults are compiled using a GCC 7.2 toolchain created using crosstools-ng and distributed as conda packages. Builds are done inside a CentOS 6 docker container.

Adding some info, as I was hunting for the actual recipes recently, and found this issue first:

• background: https://www.anaconda.com/blog/developer-blog/utilizing-the-new-compilers-in-anaconda-distribution-5/
• points here: https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html#customizing-the-compilers
• which leads to the current build recipes for the conda 3 toolchain compilers:
  • clang (mac): https://github.com/AnacondaRecipes/aggregate/tree/master/clang
  • gcc: https://github.com/AnacondaRecipes/aggregate/tree/master/ctng-compilers-feedstock (confirmed)

[mjscosta]

Any statment on this regard, on Windows (10) binary compatibility ?

[msarahan]

Win10 binary compatibility is simpler. You have classes of compatibility. It depends on which compiler gets used. Stuff compiled with VS 2008 is associated with python 2.7. VS2015 is associated with python 3.5 and up. VS2017 is compatible with VS2015, but there are some small details. If you build something with VS2017, you need the VS 2017 runtimes (called vs2015_runtime, so that they don't coexist with actual vs 2015 runtimes, because they have the same filenames). The vs 2017 runtimes work with software compiled with VS 2015. It's like the bounds for libstdc++: bundle the newest runtime, and you'll be fine. Our compiler activation packages are set up to take care of this for you.

[mjscosta]

@msarahan Thanks for you quick feedback.
I'm having some issues comming on the sorts of this ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. when importing some modules.
I'm using an environment with python 3.6 with packages from conda-forge and defaults (anaconda), and I'm trying to exclude the issue being binary compatability between both.
I'm using nuitka to compile some application code as .pyd , which migh be causing this at some point ...

[teake]

@jjhelmus Is this still the place to track progress? If so, what's the status? I've noticed that some conda-forge packages have a gcc7 label. Are these binary compatible with defaults, and those without not?

I'm asking since I'm in the process of building packages that have runtime dependencies on conda-forge-only packages. But I want to use the defaults compilers to build them. Should I then always use the gcc7 label for the runtime dependencies? Or am I better of recompiling said packages with the defaults compilers?

I'm sorry if this has already been discussed elsewhere, but unfortunately I can't see the forest through the trees at the moment.

[CJ-Wright]

Yes the packages on gcc7 should be compatible with defaults.

[pauldmccarthy]

Apologies for raising this old thread, but it seems to be the best/most canonical source of information regarding C++ ABI compatibility on conda-forge-provided packages.

Is it still the case that C++ packages on conda-forge are/should be compiled against the old ABI (i.e. with -D_GLIBCXX_USE_CXX11_ABI=0)?

I ask, because I am continually running into problems where it appears that this is not the case. For example, the latest version of boost-cpp (1.74) appears to have been compiled against the new ABI:

➜  wget https://anaconda.org/conda-forge/boost-cpp/1.74.0/download/linux-64/boost-cpp-1.74.0-h9d3c048_1.tar.bz2
...
2021-01-04 15:31:30 (1.94 MB/s) - ‘boost-cpp-1.74.0-h9d3c048_1.tar.bz2’ saved [17069586/17069586]

➜  tar xf boost-cpp-1.74.0-h9d3c048_1.tar.bz2

Now note that the functions in the filesystem library expect types of std::__cxx11::basic_string, rather than std::string:

➜  nm -gDC lib/libboost_filesystem.so|grep string|head -n2
0000000000019ed0 T boost::filesystem::path_traits::convert(char const*, char const*, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&, std::codecvt<wchar_t, char, __mbstate_t> const&)
0000000000019480 T boost::filesystem::path_traits::convert(wchar_t const*, wchar_t const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::codecvt<wchar_t, char, __mbstate_t> const&)

Am I simply misguided or misinformed? Or should I expect that certain packages on conda-forge have not been compiled with -D_GLIBCXX_USE_CXX11_ABI=0? Or should I just give up on ABI=0, and start working with packages from conda-forge/labels/gcc7?

[jjhelmus]

conda-forge completed the migration to the "new" compilers in January of 2019. These compilers use the newer C++11 ABI. Packages should no longer be compiled with the -D_GLIBCXX_USE_CXX11_ABI=0 flag.

I'm closing this issue since the migration to the new compiler removes the compatibility concerns discussed in this issue.