Split main executable and preload library

Since glibc version 2.30, using dlopen or LD_PRELOAD on PIE
objects/executables is no longer allowed and doing so will result in an
error like this:

  ERROR: ld.so: object ... from LD_PRELOAD cannot be preloaded (cannot
                dynamically load position-independent executable):
                ignored.

The upstream patch that introduced this change can be found here:

https://patchwork.ozlabs.org/patch/1055380/

Also linked over there is the comment to the bugtracker, with the
quote from https://sourceware.org/bugzilla/show_bug.cgi?id=11754#c13:

> We cannot support this because it is not possible to perform correct
> relocations if another executable has already been loaded. There is
> also no way to correctly execute the ELF constructors of the second
> executable.

> If you want to inject code into another executable, you can use
> LD_PRELOAD or LD_AUDIT, which does not have these problems.

So if I understand this correctly, we shouldn't be affected since we
first of all actually *want* to override symbols and second, we already
have a linker script that should make sure that we don't "leak" out too
many symbols.

Nevertheless however, I decided that the best way to move forward is to
split up the library and the executable since it probably would result
in less trouble on other platforms.

Unfortunately, having a separate library makes things a bit more
complicated, since the executable no longer just needs to find itself
and use itself in LD_PRELOAD.

My first attempt was to hardcode the path to the installed library
location into the source code of the executable, but this not only made
it difficult to relocate the library but we'd also need a way to specify
a different location during our tests.

The next attempt was to use utilise the runtime dynamic loader via
dlopen() to find the library. This however would also mean that we'd
actually need to know the shared library suffix and library name for the
dlopen call.

However, Meson currently doesn't seem to have a very good way to refer
to the shared library prefix and the only way to figure out the name was
to use .full_path() on the shared library. The latter however doesn't
return the *installed* location and getting just the basename would also
mean using another run_command() call during build.

Another idea I was toying around was to actually link the library to the
main executable, so that even when shrinking/stripping the rpath, both
will always be linked together in one form or another and to use
dladdr() or /proc/self/maps to determine the path from the loaded
library. Unfortunately this approach is not very portable since dladdr()
is a nonstandord GNU extension.

To make sure I'm not missing anything I went to the #mesonbuild IRC
channel to ask for best practices and @eli-schwartz managed to convince
me (thanks!) to indeed go for the last route by refering to the
following SO question:

https://stackoverflow.com/q/43409167

The comments/answers pretty much cover all the systems we're trying to
support and we probably won't ever be able to be *fully* cross- platform
since we already wrap a particular kind of socket interface.

Additionally, this solution also seems to be the one that has the least
amount of predetermined breaking points and we don't need to hardcode
anything or implement funny heuristics to search for the library.

Right now, I've only provided an implementation for GNU/Linux, since I
don't have a Darwin machine. Adding support however shouldn't be that
hard and will be coming soon.

Signed-off-by: aszlig <aszlig@nix.build>

CHANGELOG.md

@@ -4,6 +4,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog], and this project adheres to
 [Semantic Versioning].
 
+## [Unreleased]
+
+### Fixed
+- Support for glibc >= 2.30 by splitting preload library and main executable.
+
 ## [2.1.1] - 2019-09-20
 
 ### Fixed
@@ -79,6 +84,7 @@ The format is based on [Keep a Changelog], and this project adheres to
 - The initial release, which evolved from an early prototype specific to a
   certain use case into a more generic command line tool.
 
+[Unreleased]: https://github.com/nixcloud/ip2unix/compare/v2.1.1...HEAD
 [2.1.1]: https://github.com/nixcloud/ip2unix/compare/v2.1.0...v2.1.1
 [2.1.0]: https://github.com/nixcloud/ip2unix/compare/v2.0.1...v2.1.0
 [2.0.1]: https://github.com/nixcloud/ip2unix/compare/v2.0.0...v2.0.1

meson.build

@@ -26,12 +26,9 @@ add_project_arguments(cc.get_supported_arguments(warning_flags),
 
 python = import('python').find_installation('python3')
 
-cflags = ['-fPIC', '-DVERSION="' + meson.project_version() + '"']
-
-ldflags = [
-  '-pie',
-  '-Wl,-E',
-]
+cflags = ['-DVERSION="' + meson.project_version() + '"']
+main_cflags = []
+lib_cflags = []
 
 deps = [
   dependency('yaml-cpp', version: '>=0.5.0'),
@@ -42,7 +39,7 @@ libcpath = run_command(python, script_findlibc, cc.cmd_array())
 if libcpath.returncode() == 0
   fullpath = libcpath.stdout()
   message('Found C library at ' + fullpath)
-  cflags += ['-DLIBC_PATH="' + fullpath + '"']
+  lib_cflags += ['-DLIBC_PATH="' + fullpath + '"']
 endif
 
 systemd_enabled = get_option('systemd-support')
@@ -51,18 +48,25 @@ if systemd_enabled
   cflags += ['-DSYSTEMD_SUPPORT']
 endif
 
-sources = []
+lib_sources = []
+main_sources = []
 includes = []
 subdir('src')
 
 generate_sym_map = [python, script_gensyms, '@INPUT@']
-sym_map = custom_target('symmap', input: sources, output: 'symbols.map',
+sym_map = custom_target('symmap', input: lib_sources, output: 'symbols.map',
                         command: generate_sym_map, capture: true)
 cflags += ['-Wl,--version-script,@0@'.format(sym_map.full_path())]
 
-ip2unix = executable('ip2unix', sources, install: true, dependencies: deps,
-                     link_args: ldflags, link_depends: sym_map,
-                     include_directories: includes, cpp_args: cflags)
+libip2unix = shared_library('ip2unix', lib_sources, install: true,
+                            dependencies: deps, link_depends: sym_map,
+                            cpp_args: lib_cflags + cflags,
+                            include_directories: includes)
+
+ip2unix = executable('ip2unix', main_sources, install: true,
+                     link_with: libip2unix,
+                     dependencies: deps, include_directories: includes,
+                     cpp_args: main_cflags + cflags)
 
 man_input = files('README.adoc')
 

src/ip2unix.cc

@@ -2,32 +2,54 @@
 #include <climits>
 #include <cstdio>
 #include <cstdlib>
+#include <cstring>
 #include <getopt.h>
 #include <string>
 #include <unistd.h>
+#include <dlfcn.h>
 
 #include "rules.hh"
 #include "serial.hh"
 
 extern char **environ;
 
+extern "C" const char *__ip2unix__(void);
+
+static std::optional<std::string> get_preload_libpath(void)
+{
+    Dl_info info;
+
+    if (dladdr(reinterpret_cast<void*>(__ip2unix__), &info) < 0) {
+        perror("dladdr");
+        return std::nullopt;
+    }
+
+    return std::string(info.dli_fname);
+}
+
 static bool run_preload(std::vector<Rule> &rules, char *argv[])
 {
-    char self[PATH_MAX], *preload;
-    ssize_t len;
+    const char *libversion;
+    char *buf, *preload;
+    std::optional<std::string> libpath;
+
+    libversion = __ip2unix__();
 
-    if ((len = readlink("/proc/self/exe", self, sizeof(self) - 1)) == -1) {
-        perror("readlink(\"/proc/self/exe\")");
+    if (!(libpath = get_preload_libpath())) {
         return false;
     }
 
-    self[len] = '\0';
+    if (strcmp(libversion, VERSION) != 0) {
+        fprintf(stderr, "Version mismatch between preload library (%s) and"
+                        " wrapper program (%s).\n", libversion, VERSION);
+        return false;
+    }
 
     if ((preload = getenv("LD_PRELOAD")) != nullptr && *preload != '\0') {
-        std::string new_preload = std::string(self) + ":" + preload;
+        std::string new_preload = libpath.value() + ":" + preload;
         setenv("LD_PRELOAD", new_preload.c_str(), 1);
     } else {
-        setenv("LD_PRELOAD", self, 1);
+        setenv("LD_PRELOAD", libpath.value().c_str(), 1);
     }
 
     setenv("__IP2UNIX_RULES", serialise(rules).c_str(), 1);

src/meson.build

@@ -14,20 +14,25 @@ dynports_sources = [dynports, files('rng.cc')]
 serial_sources = files('serial.cc')
 globpath_sources = files('globpath.cc')
 
-sources += files('ip2unix.cc',
-                 'blackhole.cc',
-                 'logging.cc',
-                 'preload.cc',
-                 'realcalls.cc',
-                 'rules/parse.cc',
-                 'socket.cc',
-                 'sockaddr.cc',
-                 'sockopts.cc')
+common_sources = files('rules/parse.cc')
+common_sources += serial_sources
+common_sources += errnos
+
+main_sources += files('ip2unix.cc')
+main_sources += common_sources
+
+lib_sources += files('blackhole.cc',
+                     'logging.cc',
+                     'preload.cc',
+                     'realcalls.cc',
+                     'socket.cc',
+                     'sockaddr.cc',
+                     'sockopts.cc')
+lib_sources += common_sources
+
 if systemd_enabled
-  sources += files('systemd.cc')
+  lib_sources += files('systemd.cc')
 endif
-sources += dynports_sources
-sources += serial_sources
-sources += globpath_sources
-sources += errnos
+lib_sources += dynports_sources
+lib_sources += globpath_sources
 includes += include_directories('.')

src/preload.cc

@@ -68,6 +68,11 @@ static void init_rules(void)
     g_rules = std::make_shared<std::vector<Rule>>(rules.value());
 }
 
+extern "C" const char *WRAP_SYM(__ip2unix__)(void)
+{
+    return VERSION;
+}
+
 extern "C" int WRAP_SYM(socket)(int domain, int type, int protocol)
 {
     TRACE_CALL("socket", domain, type, protocol);

tests/conftest.py

@@ -1,13 +1,16 @@
 import pytest
 
 IP2UNIX = None
+LIBIP2UNIX = None
 SYSTEMD_SUPPORT = False
 SYSTEMD_SA_PATH = None
 
 
 def pytest_addoption(parser):
     parser.addoption('--ip2unix-path', action='store',
                      help='The path to the ip2unix command')
+    parser.addoption('--libip2unix-path', action='store',
+                     help='The path to the ip2unix library')
     parser.addoption('--systemd-support', action='store_true',
                      help='Whether systemd support is compiled in')
     parser.addoption('--systemd-sa-path', action='store',
@@ -23,8 +26,10 @@ def helper_accept_no_peer_addr(request):
 
 def pytest_configure(config):
     global IP2UNIX
+    global LIBIP2UNIX
     global SYSTEMD_SUPPORT
     global SYSTEMD_SA_PATH
     IP2UNIX = config.option.ip2unix_path
+    LIBIP2UNIX = config.option.libip2unix_path
     SYSTEMD_SUPPORT = config.option.systemd_support
     SYSTEMD_SA_PATH = config.option.systemd_sa_path

tests/helper.py

@@ -4,9 +4,9 @@
 from contextlib import contextmanager
 
 import pytest
-from conftest import IP2UNIX, SYSTEMD_SUPPORT, SYSTEMD_SA_PATH
+from conftest import IP2UNIX, LIBIP2UNIX, SYSTEMD_SUPPORT, SYSTEMD_SA_PATH
 
-__all__ = ['IP2UNIX', 'SYSTEMD_SUPPORT', 'SYSTEMD_SA_PATH',
+__all__ = ['IP2UNIX', 'LIBIP2UNIX', 'SYSTEMD_SUPPORT', 'SYSTEMD_SA_PATH',
            'ip2unix', 'systemd_only', 'non_systemd_only',
            'systemd_sa_helper_only']
 

tests/meson.build

@@ -6,6 +6,7 @@ if pytest.found()
   pytest_args = [
     '-p', 'no:cacheprovider',
     '--ip2unix-path=@0@'.format(ip2unix.full_path()),
+    '--libip2unix-path=@0@'.format(libip2unix.full_path()),
     '--helper-accept-no-peer-addr=@0@'.format(
       helper_accept_no_peer_addr.full_path()
     )

tests/test_program_args.py

@@ -2,7 +2,7 @@
 import sys
 import subprocess
 
-from helper import IP2UNIX
+from helper import IP2UNIX, LIBIP2UNIX
 
 
 def check_error(cmd):
@@ -76,5 +76,5 @@ def test_existing_ld_preload():
     testprog = "import os; print(os.environ['LD_PRELOAD'])"
     cmd = [IP2UNIX, '-r', 'path=/foo', sys.executable, '-c', testprog]
     output = subprocess.check_output(cmd, env={'LD_PRELOAD': '/nonexistent'})
-    expect = IP2UNIX + ":/nonexistent"
+    expect = LIBIP2UNIX + ":/nonexistent"
     assert output.decode().strip() == expect

tests/test_run_direct.py

@@ -24,7 +24,7 @@ def test_run_direct(tmpdir):
     cmd = [sys.executable, '-c', TESTPROG]
 
     env = {
-        'LD_PRELOAD': helper.IP2UNIX,
+        'LD_PRELOAD': helper.LIBIP2UNIX,
         'IP2UNIX_RULE_FILE': str(rulefile),
     }
 
@@ -41,7 +41,7 @@ def test_run_direct(tmpdir):
 
 def test_run_direct_fail():
     cmd = [sys.executable, '-c', TESTPROG]
-    env = {'LD_PRELOAD': helper.IP2UNIX}
+    env = {'LD_PRELOAD': helper.LIBIP2UNIX}
 
     with subprocess.Popen(cmd, env=env, stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE) as proc:
@@ -52,7 +52,7 @@ def test_run_direct_fail():
 
 def test_run_direct_invalid_rules():
     cmd = [sys.executable, '-c', TESTPROG]
-    env = {'LD_PRELOAD': helper.IP2UNIX, '__IP2UNIX_RULES': '{'}
+    env = {'LD_PRELOAD': helper.LIBIP2UNIX, '__IP2UNIX_RULES': '{'}
 
     with subprocess.Popen(cmd, env=env, stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE) as proc: