glibc: keg_only and optionally add system ld search paths #109998

XuehaiPan · 2022-09-08T13:14:12Z

Have you followed the guidelines for contributing?
Have you ensured that your commits follow the commit style guide?
Have you checked that there aren't other open pull requests for the same formula update/change?
Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

Add system ld search paths by including dd system /etc/ld.so.conf.

Before:

$ $(brew --prefix)/sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t'
/home/linuxbrew/.linuxbrew/Cellar/glibc/2.35/lib: (from <builtin>:0)

$ /usr/sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t'
/usr/local/cuda-11.6/targets/x86_64-linux/lib:
/usr/lib/x86_64-linux-gnu/libfakeroot:
/usr/lib/wsl/lib:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/lib32:
/libx32:
/lib:

After:

$ $(brew --prefix)/sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t'
/usr/local/cuda-11.6/targets/x86_64-linux/lib: (from /etc/ld.so.conf.d/cuda-11-6.conf:1)
/usr/lib/x86_64-linux-gnu/libfakeroot: (from /etc/ld.so.conf.d/fakeroot-x86_64-linux-gnu.conf:1)
/usr/lib/wsl/lib: (from /etc/ld.so.conf.d/ld.wsl.conf:4)
/usr/local/lib: (from /etc/ld.so.conf.d/libc.conf:2)
/lib/x86_64-linux-gnu: (from /etc/ld.so.conf.d/x86_64-linux-gnu.conf:3)
/lib32: (from /etc/ld.so.conf.d/zz_i386-biarch-compat.conf:2)
/libx32: (from /etc/ld.so.conf.d/zz_x32-biarch-compat.conf:2)
/home/linuxbrew/.linuxbrew/Cellar/glibc/2.35/lib: (from <builtin>:0)

XuehaiPan · 2022-09-08T13:30:45Z

Now the formula glibc is a dependency for all formulae on Linux (brew 3.6.0). If we don't add the system ld.so.conf, the following Python code cannot find system installed libraries:

In [1]: import ctypes

In [2]: libcuda = ctypes.CDLL('libcuda.so.1')  # in /lib/x86_64-linux-gnu
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In [2], line 1
----> 1 libcuda = ctypes.CDLL('libcuda.so.1')

File /home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/ctypes/__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371 self._FuncPtr = _FuncPtr
    373 if handle is None:
--> 374     self._handle = _dlopen(self._name, mode)
    375 else:
    376     self._handle = handle

OSError: libcuda.so.1: cannot open shared object file: No such file or directory

danielnachun · 2022-09-08T16:09:10Z

Please drop glibc@2.13 from this as it will be removed.

We need to be really careful about adding this, as mixing non-brewed libraries with brewed libraries by linkage like this can lead to all sorts of weird failures, especially if building from source.

XuehaiPan · 2022-09-08T16:25:46Z

We need to be really careful about adding this, as mixing non-brewed libraries with brewed libraries by linkage like this can lead to all sorts of weird failures, especially if building from source.

Yes, but since the glibc formula is linked to HOMEBREW_PREFIX, it breaks many things. E.g., ldconfig in HOMEBREW_PREFIX/sbin (in user's PATH) breaks Python's ctypes module. See also:

Having HOMEBREW_PREFIX/lib in LD_LIBRARY_PATH breaks things when glibc installed #109893
linux/diagnostic: check LD_LIBRARY_PATH when glibc is installed. brew#13826

I think we should make glibc a keg-only formula.

danielnachun · 2022-09-08T18:43:09Z

Making it keg-only will also break many things at the moment so we will have to find a suitable compromise solution. I will do some local testing of this to figure out what the best solution, but my general inclination is to say that mixing brewed and system libraries when using brewed glibc is going to be supported on a "best effort" basis. We should prioritize making sure things work properly with brewed libraries as a higher priority than working with system libraries.

XuehaiPan · 2022-09-09T08:48:28Z

Brewed Python cannot import packages that ship shared libraries (e.g. libtorch*.so in torch), even in a virtual environment:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install --upgrade pip setuptools rich
$ pip3 install --upgrade torch

$ python3 -c 'from rich import traceback; traceback.install(); import torch'
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <string>:1 in <module>                                                                           │
│                                                                                                  │
│ /home/PanXuehai/venv/lib/python3.10/site-packages/torch/__init__.py:201 in <module>              │
│                                                                                                  │
│   198 │   #                                                                                      │
│   199 │   # See Note [Global dependencies]                                                       │
│   200 │   if USE_GLOBAL_DEPS:                                                                    │
│ ❱ 201 │   │   _load_global_deps()                                                                │
│   202 │   from torch._C import *  # noqa: F403                                                   │
│   203                                                                                            │
│   204 # Appease the type checker; ordinarily this binding is inserted by the                     │
│                                                                                                  │
│ /home/PanXuehai/venv/lib/python3.10/site-packages/torch/__init__.py:154 in _load_global_deps     │
│                                                                                                  │
│   151 │   here = os.path.abspath(__file__)                                                       │
│   152 │   lib_path = os.path.join(os.path.dirname(here), 'lib', lib_name)                        │
│   153 │                                                                                          │
│ ❱ 154 │   ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)                                         │
│   155                                                                                            │
│   156                                                                                            │
│   157 if (USE_RTLD_GLOBAL_WITH_LIBTORCH or os.getenv('TORCH_USE_RTLD_GLOBAL')) and \             │
│                                                                                                  │
│ /home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/ctypes/__init__.py:374 in __init__     │
│                                                                                                  │
│   371 │   │   self._FuncPtr = _FuncPtr                                                           │
│   372 │   │                                                                                      │
│   373 │   │   if handle is None:                                                                 │
│ ❱ 374 │   │   │   self._handle = _dlopen(self._name, mode)                                       │
│   375 │   │   else:                                                                              │
│   376 │   │   │   self._handle = handle                                                          │
│   377                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OSError: libstdc++.so.6: cannot open shared object file: No such file or directory

We do have a libstdc++.so.6 linked to HOMEBREW_PREFIEX.

$ fd '^libstdc++' "${HOMEBREW_PREFIX}"
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++.a
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++.so
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++.so.6
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++.so.6.0.30
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++.so.6.0.30-gdb.py
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/12/libstdc++fs.a
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++.a
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++.so
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++.so.6
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++.so.6.0.30
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++.so.6.0.30-gdb.py
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/lib/gcc/current/libstdc++fs.a
/home/linuxbrew/.linuxbrew/Cellar/gcc/12.2.0/share/gcc-12/python/libstdcxx
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++.a
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++.so
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++.so.6
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++.so.6.0.29
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++.so.6.0.29-gdb.py
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/lib/gcc/11/libstdc++fs.a
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0/share/gcc-11/python/libstdcxx
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++.a
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++.so
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++.so.6
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++.so.6.0.29
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++.so.6.0.29-gdb.py
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/lib/gcc/11/libstdc++fs.a
/home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.3.0.reinstall/share/gcc-11/python/libstdcxx
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++.a
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++.so
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++.so.6
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++.so.6.0.30
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++.so.6.0.30-gdb.py
/home/linuxbrew/.linuxbrew/lib/gcc/12/libstdc++fs.a
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++.a
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++.so
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++.so.6
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++.so.6.0.30
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++.so.6.0.30-gdb.py
/home/linuxbrew/.linuxbrew/lib/gcc/current/libstdc++fs.a
/home/linuxbrew/.linuxbrew/lib/libstdc++.so.6

danielnachun · 2022-09-09T16:11:18Z

The problem here is that libtorch.so is a precompiled library built outside of Homebrew that is using the host linker and glibc. We go to great pains to avoid linking to any host libraries other than glibc and some GCC runtime libraries if they are new enough to avoid exactly these kinds of issues. Similarly we don't allow packages to include pre-compiled binaries because they can have similar problems with not being able to find the proper libraries. I am going to test the effects of making glibc keg-only but even if this is possible it's not going to fix all of these issues. The reality is that mixing brewed and host libraries like this is inherently messy and prone to failure.

It's worth reiterating here that the intended use case for brewed Python is really as a run time for Python applications rather than as a environment into which the user installs their own packages. Users should really use virtualenvs directly from Python or from Miniconda/Anaconda when they want to do things like install Python packages with pip that depend on native libraries built outside of Homebrew. That it is sometimes possible to install Python packages that depend on native libraries in brewed Python with pip is just a matter of luck rather than something that is supported.

Formula/glibc.rb

sjackman · 2022-09-12T21:06:28Z

@XuehaiPan Is upgrading to a distribution of Linux like Ubuntu 22.04 that provides Glibc 2.35 or newer an option for you? You wouldn't need to install the brewed glibc if so.

danielnachun · 2022-09-12T21:45:41Z

I agree that users should upgrade when possible, but I'm warming up to this idea. I want to test it locally in the next day. One note: we need to drop all explicit glibc dependencies in homebrew-core because glibc is now handled as a global dependency that is handled by brew rather than specific formulae. For now @XuehaiPan you can drop the revision bumps and I have set this to skip dependent testing because it should not actually be needed..

XuehaiPan · 2022-09-13T06:29:15Z

@XuehaiPan Is upgrading to a distribution of Linux like Ubuntu 22.04 that provides Glibc 2.35 or newer an option for you? You wouldn't need to install the brewed glibc if so.

I'm afraid not. Most of my use cases for Homebrew on Linux are installing packages on Linux servers where I don't have sudo privileges. I think another reason people use Homebrew on Linux is to have both benefits from the system-provided stable production environment (for me, NVIDIA drivers and CUDA Toolkit) and the latest command line software from Homebrew (bat, ripgrep, fd, etc.).

Since the Ubuntu LTS versions provide 5 years long term support, the users are not meant to upgrade their system each time a new LTS version is released. Especially, the system is shared for multiple users, e.g., cloud services or ML servers.

In Support Policy of GitHub Actions Runner and GitHub-hosted Runner Matrix, GitHub provides at least two GA version of an OS. And, currently ubuntu-latest is ubuntu-20.04.

Note: The -latest runner images are the latest stable images that GitHub provides, and might not be the most recent version of the operating system available from the operating system vendor. From GitHub-hosted Runner Matrix

As discussed in Why always use the latest version?:

Homebrew is a rolling-release package manager. We try to ship the newest things as quickly as possible, on macOS and Linux.

we always use the latest LTS version of Ubuntu to build bottles. But I wonder could we use the second latest LTS version (20.04 for now) or sync the same version of ubuntu-latest on GitHub Action runner images? The system package wouldn't be too old and that will not make Homebrew rolls release too aggressively.

danielnachun · 2022-09-13T17:12:13Z

we always use the latest LTS version of Ubuntu to build bottles. But I wonder could we use the second latest LTS version (20.04 for now) or sync the same version of ubuntu-latest on GitHub Action runner images? The system package wouldn't be too old and that will not make Homebrew rolls release too aggressively.

We decided against this for a number of reasons. Our goal right now is to fix the underlying issues with brewed glibc so that it "just works" as intended. I think we are getting close to this now and this contribution is a step in the right direction.

Formula/glibc.rb

sjackman · 2022-09-13T18:43:43Z

Making glibc keg-only seems like a good idea (I think). It would address #109893. Adding glibc.lib to the ld.so.conf search path would address finding the glibc libraries at run time. We would then need to address finding the glibc include and library files at build time, which seems quite doable using either GCC specs or superenv or both.

Having HOMEBREW_PREFIX/lib in LD_LIBRARY_PATH breaks things when glibc installed #109893

sjackman · 2022-09-13T18:47:12Z

This change opens a hole can of worms mixing Homebrew shared libraries with host shared libraries which use different versions of Glibc. I expect there's fun ways it could breaks, but it also addresses real issues that arise primarily when trying to compile things from source outside of Homebrew, particularly installing packages for other programming languages using their package manager and building native extensions from source. It's certainly worth experimenting with this PR.

A small historical note: Using patchelf on shared libraries used to break ldconfig, which is why Homebrew on Linux has never used ldconfig. I suspect it was a Glibc bug. I'm really happy to see that that bug has been fixed, and we can use ld.so.conf and ld.so.cache now.

Formula/glibc.rb

Also: * Make linux-headers@5.15 a runtime dependency * Don't link ld.so in post-install (instead rely on behaviour within brew) * Use correct etc directory * Architecture-agnostic test * Ensure /tmp/homebrew is always cleaned Co-authored-by: Bo Anderson <mail@boanderson.me>

XuehaiPan · 2022-09-19T05:46:23Z

Failed to build gcc and pkg-config, does this PR require Homebrew/brew#13873 to be merged first?

Bo98 · 2022-09-19T05:46:42Z

Yes.

Bo98 · 2022-09-19T06:33:29Z

$ brew install -s pkg-config
==> Downloading https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/linuxbrew/.linuxbrew/Cellar/pkg-config/0.29.2_3 --disable-host-tool --with-internal-glib --with-pc-path=/home/linuxbrew/.linuxbrew/lib/pkgconfig:/home/linuxbrew/.linuxbrew/share/pkgconfig:/home/linuxbrew/.li
==> make
==> make install
🍺  /home/linuxbrew/.linuxbrew/Cellar/pkg-config/0.29.2_3: 11 files, 742.2KB, built in 51 seconds
==> Running `brew cleanup pkg-config`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Bo98

Thanks

BrewTestBot · 2022-09-19T07:12:22Z

@Bo98 has triggered a merge.

BrewTestBot added the linux-only Formula depends on Linux label Sep 8, 2022

XuehaiPan force-pushed the glibc-ld.so.conf branch from c45a7c5 to 01291c9 Compare September 8, 2022 13:52

Bo98 requested a review from a team September 8, 2022 16:37

XuehaiPan force-pushed the glibc-ld.so.conf branch 2 times, most recently from bbc5ddd to d4863d3 Compare September 9, 2022 04:46

XuehaiPan force-pushed the glibc-ld.so.conf branch from d4863d3 to be90dd4 Compare September 10, 2022 11:34

XuehaiPan commented Sep 10, 2022

View reviewed changes

Formula/glibc.rb Show resolved Hide resolved

XuehaiPan force-pushed the glibc-ld.so.conf branch from be90dd4 to 78eb64f Compare September 11, 2022 16:06

danielnachun added CI-skip-recursive-dependents Pass --skip-recursive-dependents to brew test-bot. CI-skip-dependents Pass --skip-dependents to brew test-bot. labels Sep 12, 2022

XuehaiPan force-pushed the glibc-ld.so.conf branch from 78eb64f to 2ca58b0 Compare September 13, 2022 05:25

danielnachun reviewed Sep 13, 2022

View reviewed changes

Formula/glibc.rb Outdated Show resolved Hide resolved

sjackman mentioned this pull request Sep 13, 2022

Having HOMEBREW_PREFIX/lib in LD_LIBRARY_PATH breaks things when glibc installed #109893

Closed

2 tasks

sjackman requested review from iMichka, Bo98, carlocab, fxcoudert and SMillerDev September 13, 2022 18:53

carlocab reviewed Sep 17, 2022

View reviewed changes

Formula/glibc.rb Outdated Show resolved Hide resolved

Formula/glibc.rb Outdated Show resolved Hide resolved

Formula/glibc.rb Outdated Show resolved Hide resolved

XuehaiPan force-pushed the glibc-ld.so.conf branch 6 times, most recently from 6b06dc9 to 1296c4f Compare September 17, 2022 16:19

Bo98 force-pushed the glibc-ld.so.conf branch from 1296c4f to ff34880 Compare September 19, 2022 02:24

carlocab previously approved these changes Sep 19, 2022

View reviewed changes

Formula/glibc.rb Outdated Show resolved Hide resolved

Bo98 dismissed carlocab’s stale review via bee32b9 September 19, 2022 03:32

Bo98 force-pushed the glibc-ld.so.conf branch from ff34880 to bee32b9 Compare September 19, 2022 03:32

Bo98 added the automerge-skip `brew pr-automerge` will skip this pull request label Sep 19, 2022

XuehaiPan force-pushed the glibc-ld.so.conf branch from bee32b9 to b164c44 Compare September 19, 2022 04:42

BrewTestBot removed the automerge-skip `brew pr-automerge` will skip this pull request label Sep 19, 2022

Bo98 added the automerge-skip `brew pr-automerge` will skip this pull request label Sep 19, 2022

XuehaiPan force-pushed the glibc-ld.so.conf branch from b164c44 to fe7d8cb Compare September 19, 2022 05:12

BrewTestBot removed the automerge-skip `brew pr-automerge` will skip this pull request label Sep 19, 2022

Bo98 added the automerge-skip `brew pr-automerge` will skip this pull request label Sep 19, 2022

Bo98 approved these changes Sep 19, 2022

View reviewed changes

BrewTestBot closed this in d08334b Sep 19, 2022

sjackman changed the title ~~glibc: add system ld search paths~~ glibc: keg_only and optionally add system ld search paths Sep 21, 2022

sjackman mentioned this pull request Sep 21, 2022

linux/diagnostic: check LD_LIBRARY_PATH when glibc is installed. Homebrew/brew#13826

Closed

7 tasks

XuehaiPan deleted the glibc-ld.so.conf branch October 15, 2022 12:52

github-actions bot added the outdated PR was locked due to age label Nov 15, 2022

github-actions bot locked as resolved and limited conversation to collaborators Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

glibc: keg_only and optionally add system ld search paths #109998

glibc: keg_only and optionally add system ld search paths #109998

XuehaiPan commented Sep 8, 2022

XuehaiPan commented Sep 8, 2022

danielnachun commented Sep 8, 2022

XuehaiPan commented Sep 8, 2022 •

edited

danielnachun commented Sep 8, 2022

XuehaiPan commented Sep 9, 2022 •

edited

danielnachun commented Sep 9, 2022

sjackman commented Sep 12, 2022 •

edited

danielnachun commented Sep 12, 2022

XuehaiPan commented Sep 13, 2022

danielnachun commented Sep 13, 2022

sjackman commented Sep 13, 2022

sjackman commented Sep 13, 2022

XuehaiPan commented Sep 19, 2022

Bo98 commented Sep 19, 2022

Bo98 commented Sep 19, 2022

Bo98 left a comment

BrewTestBot commented Sep 19, 2022

glibc: keg_only and optionally add system ld search paths #109998

glibc: keg_only and optionally add system ld search paths #109998

Conversation

XuehaiPan commented Sep 8, 2022

XuehaiPan commented Sep 8, 2022

danielnachun commented Sep 8, 2022

XuehaiPan commented Sep 8, 2022 • edited

danielnachun commented Sep 8, 2022

XuehaiPan commented Sep 9, 2022 • edited

danielnachun commented Sep 9, 2022

sjackman commented Sep 12, 2022 • edited

danielnachun commented Sep 12, 2022

XuehaiPan commented Sep 13, 2022

danielnachun commented Sep 13, 2022

sjackman commented Sep 13, 2022

sjackman commented Sep 13, 2022

XuehaiPan commented Sep 19, 2022

Bo98 commented Sep 19, 2022

Bo98 commented Sep 19, 2022

Bo98 left a comment

Choose a reason for hiding this comment

BrewTestBot commented Sep 19, 2022

XuehaiPan commented Sep 8, 2022 •

edited

XuehaiPan commented Sep 9, 2022 •

edited

sjackman commented Sep 12, 2022 •

edited