Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

meson does not properly compute RUNPATH on a merged-usr system #4392

Closed
mbiebl opened this issue Oct 19, 2018 · 23 comments
Closed

meson does not properly compute RUNPATH on a merged-usr system #4392

mbiebl opened this issue Oct 19, 2018 · 23 comments

Comments

@mbiebl
Copy link

mbiebl commented Oct 19, 2018

This was originally filed as systemd/systemd#10430

Building systemd git master on a merged-/usr Debian sid system, results in broken test binaries.
The RUNPATH is not properly set. After applying #4381, the test binaries are usable.
But in case of /lib/x86_64-linux-gnu, it shouldn't be added to RUNPATH at all, because this is a system library path.
On a non-merged-/usr system, no such RUNPATH is added. It seems that the lib -> usr/lib symlink breaks the system library detection.
@mbiebl mbiebl mentioned this issue Oct 19, 2018
Closed
@mbiebl
Copy link
Author

mbiebl commented Oct 19, 2018

A merged-/usr chroot can easily be created on a Debian sid system.
debootstrap in sid defaults to --merged-usr, if you are using debootstrap from stretch, you need to explicitly add this command line option.

The result is a chroot which looks like this

# ls -la /
total 280
drwxr-xr-x  18 root root   4096 Oct 19 19:42 .
drwxr-xr-x  18 root root   4096 Oct 19 19:42 ..
-rw-r--r--   1 root root 201280 Oct 19 19:42 .ilist
lrwxrwxrwx   2 root root      7 Jun 14 22:39 bin -> usr/bin                                                        
drwxr-xr-x   2 root root   4096 Jan 13  2018 boot
drwxr-xr-x   2 root root   4096 Jun 14 22:40 build
drwxr-xr-x   4 root root   4096 Oct 19 19:42 dev
drwxr-xr-x  42 root root   4096 Oct 19 19:44 etc
drwxr-xr-x   2 root root   4096 Jan 13  2018 home
lrwxrwxrwx   2 root root      7 Jun 14 22:39 lib -> usr/lib                                                        
lrwxrwxrwx   2 root root      9 Jun 14 22:39 lib32 -> usr/lib32                                                    
lrwxrwxrwx   2 root root      9 Jun 14 22:39 lib64 -> usr/lib64                                                    
lrwxrwxrwx   2 root root     10 Jun 14 22:39 libx32 -> usr/libx32                                                  
drwxr-xr-x   2 root root   4096 Jun 14 22:39 media
drwxr-xr-x   2 root root   4096 Jun 14 22:39 mnt
drwxr-xr-x   2 root root   4096 Jun 14 22:39 opt
dr-xr-xr-x 294 root root      0 Oct 19 06:38 proc
drwx------   2 root root   4096 Oct 19 19:58 root
drwxr-xr-x   4 root root   4096 Oct 19 19:56 run
lrwxrwxrwx   2 root root      8 Jun 14 22:39 sbin -> usr/sbin                                                      
drwxr-xr-x   2 root root   4096 Jun 14 22:39 srv
dr-xr-xr-x  13 root root      0 Oct 19 19:58 sys
drwxrwxrwt   5 root root  20480 Oct 19 19:58 tmp
drwxr-xr-x  13 root root   4096 Jun 14 22:39 usr
drwxr-xr-x  11 root root   4096 Jun 14 22:39 var

The ld.so.config

# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

# cat /etc/ld.so.conf.d/libc.conf 
# libc default configuration
/usr/local/lib

# cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf 
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu

@mbiebl
Copy link
Author

mbiebl commented Oct 19, 2018

happy to provide more information if needed

@jpakkane
Copy link
Member

Can you copypaste the linker command Meson uses to generate a program with a broken rpath?

@jpakkane
Copy link
Member

I managed to finally debug why this happens. And it is symlinks, yet again.

In c.py method get_library_dirs we determine what are the system's library paths. Gcc returns a list containing /lib/x86_64-linux-gnu. However because many of the other paths it returns contain garbage like /usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib we normalise the path with Pahtlib. What it also does is resolve all symlinks, so the path now becomes /usr/lib/x86_64-linux-gnu, meaning that /lib is not stored as a system link path. Since (probably) library detection returns paths given by GCC and it claims that the libraries' path is /lib/x86_64-linux-gnu we don't consider it to be a "system library path" and add it to rpath.

If all symlinks pointing to library directories would be removed, the problem would go away. Obviously this is not something distros can do particularly fast so that's out. So then there's questions like:

  • Is it always ok to to normalize library paths from /lib/whatever to the symlinkless path or is there some legacy thing somewhere that breaks?
  • Why does GCC return paths that are symlinks to somewhere else? Will they be removed at some point?
  • Is there a bug somewhere in our dependency resolver that converts cc.find_library('m') and other libs to (presumably) /lib/x86_64-linux-gnu/libm.so? Should we unsymlink them? See also the first question?

@mbiebl
Copy link
Author

mbiebl commented Nov 13, 2018

running ld --verbose | grep SEARCH_DIR in a merged-usr chroot yields

# ld --verbose | grep SEARCH_DIR
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib");

in a unmerged-usr system

# ld --verbose | grep SEARCH_DIR 
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib");

@jpakkane
Copy link
Member

We get them via the compiler (we never invoke the linker directly) with this command gcc --print-search-dirs. Its output is very different and long but I don't have a Linux machine at hand to copypaste from.

@QuLogic
Copy link
Member

QuLogic commented Nov 13, 2018

On Fedora, which has had usrmerge for a while, the output is:

$ gcc --print-search-dirs
install: /usr/lib/gcc/x86_64-redhat-linux/8/
programs: =/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin/
libraries: =/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/lib/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/:/lib/x86_64-redhat-linux/8/:/lib/../lib64/:/usr/lib/x86_64-redhat-linux/8/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/lib/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../:/lib/:/usr/lib/

$ ll /
total 64
lrwxrwxrwx.   1 root root     7 Feb  7  2018 bin -> usr/bin
lrwxrwxrwx.   1 root root     7 Feb  7  2018 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Feb  7  2018 lib64 -> usr/lib64
lrwxrwxrwx.   1 root root     8 Feb  7  2018 sbin -> usr/sbin

That is, gcc outputs both /lib and /usr/lib paths.

@mbiebl
Copy link
Author

mbiebl commented Nov 13, 2018

merged-usr chroot

# 
# gcc --print-search-dirs
install: /usr/lib/gcc/x86_64-linux-gnu/8/
programs: =/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/
libraries: =/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib/:/lib/x86_64-linux-gnu/8/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/8/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../:/lib/:/usr/lib/

unmerged-usr chroot

# gcc --print-search-dirs
install: /usr/lib/gcc/x86_64-linux-gnu/8/
programs: =/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin/
libraries: =/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib/:/lib/x86_64-linux-gnu/8/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/8/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../:/lib/:/usr/lib/

In short: the output is identical

jpakkane added a commit that referenced this issue Nov 14, 2018
@jpakkane
Store unexpanded library directory paths. Closes #4392.
c658427
@jpakkane
Copy link
Member

Can you test if the linked MR fixes things for you?

jpakkane added a commit that referenced this issue Nov 14, 2018
@jpakkane
Store unexpanded library directory paths. Closes #4392.
2ba68c2
@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018 •
edited

@jpakkane looks good. Built systemd in a merged-usr environment and an unmerged-usr environment and then ran diffoscope on the results. No differences in the binaries were found.

@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018

Would be awesome to have this fix in the 0.48 branch as well, in case you are planning another 0.48.x release.

@jpakkane
Copy link
Member

There won't be a release in 0.48 unless some major showstopper bug is reported in the wild. 0.49.0 will be released December 9th.

@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018

Could we make a debian-only release then with just this commit backported?

@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018

I'm happy to NMU or sponsor the upload

@jpakkane
Copy link
Member

I updated the packaging for 0.48.2 in Mentors: https://mentors.debian.net/package/meson

For some reason Martin has not uploaded 0.48.2 yet. If you test that and find that it works (tests will fail because Rustc is still broken in Sid), feel free to upload it.

@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018

will have a look. Thanks a lot, @jpakkane

@mbiebl
Copy link
Author

mbiebl commented Nov 15, 2018

the rustc/llvm-7 issue is supposed to be fixed, let's see if that is true....

@mbiebl
Copy link
Author

mbiebl commented Nov 16, 2018

Seems to have built fine:
https://buildd.debian.org/status/package.php?p=meson

Thanks again @jpakkane

@naav
Copy link

naav commented Dec 7, 2018 •
edited

@jpakkane This change seems to break (at least) OpenSSL on my system, but possibly others that rely on meson.get_compiler('cpp'), although I haven't investigated further at the moment.

Steps to reproduce:

  • Create an LXC container (or debootstrap chroot), distribution Debian Stretch x64, configure it to have internet access if needed. Enter the container/chroot.
# Add sid repos
cat << EOF > /etc/apt/sources.list.d/unstable.list
deb http://deb.debian.org/debian sid main
EOF

# Set package priorities:
cat << EOF > /etc/apt/preferences.d/pin-preferences
Package: *
Pin: release a=stable
Pin-Priority: 900

Package: *
Pin: release a=unstable
Pin-Priority: 450
EOF

# Install stuff
apt update
apt install -y pkg-config libssl-dev g++
apt install -y meson/unstable

# Prepare dirs
mkdir src build

cat << EOF > src/meson.build
project('4392', 'cpp')
d = dependency('openssl')
EOF

meson src build

meson (0.48.2-1) gives the following output:

The Meson build system
Version: 0.48.2
Source dir: /root/src
Build dir: /root/build
Build type: native build
Project name: 4392
Project version: undefined
Native C++ compiler: c++ (gcc 6.3.0 "c++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (0.29)
Dependency openssl found: NO 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/mesonbuild/mesonmain.py", line 364, in run
    app.generate()
  File "/usr/lib/python3/dist-packages/mesonbuild/mesonmain.py", line 136, in generate
    self._generate(env)
  File "/usr/lib/python3/dist-packages/mesonbuild/mesonmain.py", line 165, in _generate
    intr.run()
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreter.py", line 3836, in run
    super().run()
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 380, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 404, in evaluate_codeblock
    raise e
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 398, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 411, in evaluate_statement
    return self.assignment(cur)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 946, in assignment
    value = self.evaluate_statement(node.value)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 409, in evaluate_statement
    return self.function_call(cur)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 705, in function_call
    return func(node, posargs, kwargs)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 273, in wrapped
    return f(*wrapped_args, **wrapped_kwargs)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 273, in wrapped
    return f(*wrapped_args, **wrapped_kwargs)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreterbase.py", line 162, in wrapped
    return f(*wrapped_args, **wrapped_kwargs)
  File "/usr/lib/python3/dist-packages/mesonbuild/interpreter.py", line 2998, in func_dependency
    dep = dependencies.find_external_dependency(name, self.environment, kwargs)
  File "/usr/lib/python3/dist-packages/mesonbuild/dependencies/base.py", line 1455, in find_external_dependency
    raise pkg_exc
  File "/usr/lib/python3/dist-packages/mesonbuild/dependencies/base.py", line 1409, in find_external_dependency
    d = c()
  File "/usr/lib/python3/dist-packages/mesonbuild/dependencies/base.py", line 529, in __init__
    self._set_libs()
  File "/usr/lib/python3/dist-packages/mesonbuild/dependencies/base.py", line 739, in _set_libs
    self.link_args, self.raw_link_args = self._search_libs(out, out_raw)
  File "/usr/lib/python3/dist-packages/mesonbuild/dependencies/base.py", line 668, in _search_libs
    libpaths, libtype)
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 999, in find_library
    return self.find_library_impl(libname, env, extra_dirs, code, libtype)
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 989, in find_library_impl
    value = self.find_library_real(libname, env, extra_dirs, code, libtype)
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 967, in find_library_real
    for d in itertools.chain(extra_dirs, self.get_library_dirs(env)):
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 249, in get_library_dirs
    return self.get_compiler_dirs(env, 'libraries')
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 244, in get_compiler_dirs
    return CCompiler._split_fetch_real_dirs(line.split('=', 1)[1])
  File "/usr/lib/python3/dist-packages/mesonbuild/compilers/c.py", line 229, in _split_fetch_real_dirs
    resolved = Path(p).resolve().as_posix()
  File "/usr/lib/python3.5/pathlib.py", line 1109, in resolve
    s = self._flavour.resolve(self)
  File "/usr/lib/python3.5/pathlib.py", line 330, in resolve
    return _resolve(base, str(path)) or sep
  File "/usr/lib/python3.5/pathlib.py", line 315, in _resolve
    target = accessor.readlink(newpath)
  File "/usr/lib/python3.5/pathlib.py", line 422, in readlink
    return os.readlink(path)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/x86_64-linux-gnu'

@nirbheek
Copy link
Member

nirbheek commented Dec 7, 2018

@naav this is already fixed with #4524, so you'll need to also include that patch.

@mbiebl
Copy link
Author

mbiebl commented Dec 7, 2018

I'm happy to sponsor an upload if @jpakkane want's to prepare an updated package with that patch included

@jpakkane
Copy link
Member

jpakkane commented Dec 7, 2018

The release is the day after tomorrow and it will fix all of these. Probably not worth the effort to do a patch upload in the mean time.

@mbiebl
Copy link
Author

mbiebl commented Dec 7, 2018

Agreed. Thanks for the heads-up.

tbeloqui pushed a commit to pexip/meson that referenced this issue Aug 22, 2019
@jpakkane @tbeloqui
Store unexpanded library directory paths. Closes mesonbuild#4392.
a6470e2
@U2FsdGVkX1 U2FsdGVkX1 mentioned this issue Feb 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nirbheek @naav @QuLogic @mbiebl @jpakkane