Identify statically-linked symbols that are needed by modules #3683

bonzini · 2018-06-04T13:52:48Z

Even though shared modules are more or less incompatible with statically linked binaries (glibc's support for libdl inside statically linked binaries, for example, is very limited), it is of course possible to link static libraries into a binary that uses shared modules. These could be either external packages, or libraries that are internal to the program being built.

Whenever an executable is statically linked with a library, the linker omits library functions that are not used by the executable. However, some of these omitted library functions, while not used by the executable, might be needed by shared modules. When this happens, the shared module will fail to load.

Because the library is statically linked, adding it to the module may not do the right thing. The most common workaround in that case is to use link_whole, but that might seriously bloat the executable and might not even always work. This pull request implements an alternative solution. It is inspired by the operation of QEMU's Makefiles, which I'm looking at for a conversion to Meson.

Module object files are processed with "nm" to find undefined symbols, and the undefined symbols are included in the executable through the linker's "-u" option. There exists equivalents in the MSVC world for both steps, but they are not implemented yet. The process is automatic and guided entirely by syntax that is already present in the Meson language.

Note that, unlike symbolextractor, this is not just an optimization. However, if for example nm is not found in the toolchain file for cross-compilation, behavior is unchanged compared to before this pull request: a module that has dependencies on a statically-linked library will fail to link, but apart from this special case everything will work normally.

ignatenkobrain · 2018-06-04T13:53:53Z

mesonbuild/scripts/undefsymbols.py

+# libraries into the main program.
+
+import os, sys
+from .. import mesonlib


[Flake8]

[F401] '..mesonlib' imported but unused

ignatenkobrain · 2018-06-04T13:53:54Z

mesonbuild/scripts/undefsymbols.py

+# touched. This information is used to include functions from static
+# libraries into the main program.
+
+import os, sys


[Flake8]

[F401] 'os' imported but unused

ignatenkobrain · 2018-06-05T12:38:02Z

mesonbuild/backend/ninjabackend.py

-            self.generate_shsym(outfile, target)
+            self.generate_shsym(rust + crstr, outfile, target)
+        if isinstance(target, build.SharedModule):
+            self.generate_undefsym(rust + crst, outfile, target)


[Flake8]

[F821] undefined name 'rust'

ignatenkobrain · 2018-06-05T12:38:03Z

mesonbuild/backend/ninjabackend.py

@@ -1332,7 +1326,9 @@ def generate_rust_target(self, target, outfile):
        element.add_item('cratetype', cratetype)
        element.write(outfile)
        if isinstance(target, build.SharedLibrary):
-            self.generate_shsym(outfile, target)
+            self.generate_shsym(rust + crstr, outfile, target)


[Flake8]

[F821] undefined name 'rust'

ignatenkobrain · 2018-06-05T13:00:02Z

mesonbuild/backend/ninjabackend.py

-            self.generate_shsym(outfile, target)
+            self.generate_shsym('rust' + crstr, outfile, target)
+        if isinstance(target, build.SharedModule):
+            self.generate_undefsym('rust' + crst, outfile, target)


[Flake8]

[F821] undefined name 'crst'

mesonbuild/scripts/undefsymbols.py

+# libraries into the main program.
+
+import sys
+from ..mesonlib import Popen_safe


nirbheek · 2018-06-05T13:20:06Z

mesonbuild/compilers/compilers.py

@@ -49,7 +49,7 @@
 c_suffixes = lang_suffixes['c'] + ('h',)
 # List of languages that can be linked with C code directly by the linker
 # used in build.py:process_compilers() and build.py:get_dynamic_linker()
-clike_langs = ('d', 'objcpp', 'cpp', 'objc', 'c', 'fortran',)
+clike_langs = ('d', 'objcpp', 'cpp', 'objc', 'c', 'fortran', 'vala',)


Please open a separate PR for this, it's bad practice to mix commits from unrelated changes together. It also means that your work will get pushed faster.

Please include the sentence "Fixes #791" in your commit message so the issue is closed automatically.

Ok, I was planning to do so but I wasn't sure how to get CI to work correctly on this PR. Thanks!

bonzini · 2018-06-15T06:15:32Z

I think I will reimplement this after adding the above feature to Ninja. However the main ideas (scripts, rules) will be the same.

jpakkane · 2018-06-17T21:25:22Z

but that might seriously bloat the executable

Do you have an estimate for how much bloat this adds?

and might not even always work

When would this happen? And also, how would linking a subset of the library then work in the same case?

bonzini · 2018-06-18T14:08:12Z

Do you have an estimate for how much bloat this adds?

For one of the binaries built by QEMU, which only links with glib (libglib-2.0.a and libgthread-2.0.a) and glibc:

qemu-x86_64 with statically linked glib: text + data = 5946217
qemu-x86_64 with whole-archive glib: text + data = 6709897

So the extra size is about 12% of the binary. The space taken by glib itself in the binary grows by a factor of 2, from ~630 KiB to ~1380 KiB.

and might not even always work

When would this happen?

In the case of QEMU, there is a "utility functions" static library used by both the emulators and the other smaller programs distributed with QEMU. Whenever some functionality is only required in the emulators, the static library includes stubs for use in the smaller tools, while the emulators override the stubs. whole_archive would cause duplicate definitions in the emulators, and linking would fail.

In the past we tried other mechanisms such as weak symbols, but static libraries turned out to be the easiest and most portable way to implement this kind of stubbing.

These are needed so that the correct set of files from static libraries is included in the executable. At build time, ninja will first collect undefined symbols from the modules, then build the executable and the import library, then finally link the module.

Instead of passing a $CROSS variable in the "build" directive, define separate rules. This introduces some duplication in the rules, but removes it from the targets because the --cross-host=... option depends only on whether the target is native or cross-compiled. Previously, it was specified via $CROSS once per shared library or module. While at it, extract the code function to generate SHSYM rules into a separate function, and generalize the function that generates SHSYM "build" directives. We will use them shortly to generate a list of undefined symbols in the static library. That usage in turn will actually require to have separate rules for each compiler.

Even though shared modules are more or less incompatible with statically linked binaries, it is of course possible to link static libraries into a binary that uses shared modules. Whenever an executable is statically linked with a library, the linker omits library functions that are not used by the executable. However, this could include functions that are not used by the executable, but are needed by shared modules. When this happens, the shared module will fail to load. One common workaround in that case is to use link_whole, but that is not always possible and might seriously bloat the executable. This commit implements an alternative solution, inspired by the operation of QEMU's Makefiles. Module object files are processed (for example with "nm") to find undefined symbols, and the undefined symbols are included in the executable through the linker's "-u" option. (There exists equivalents in the MSVC world for both steps, even though they are not implemented yet). This commit adds the logic to the generic code and to the ninja backend. The next commit is where we actually start to use "nm". Note that, unlike symbolextractor, this is not just an optimization. However, if for example nm is not found in the toolchain file for cross-compilation, behavior is unchanged compared to before this series: a module that has dependencies on a statically-linked library will fail to link, but everything else will work normally.

Finally, this patch adds support to undefsymbols for the nm utility.

Linker option -u is incompatible with -bitcode_bundle. Recognize this and avoid using the .undef files on the linker command line. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

bonzini · 2019-02-07T15:41:23Z

Looks like all CI tests pass now. Sorry for the noise!

jpakkane · 2019-02-21T22:56:10Z

Am I correct in reading the code that this is to adding a link argument like -Wl,-u,foofunc for all the missing functions?

I'm concerned about adding something automatic like this for what is a fairly niche use case. One of the things we've learned that changes like these always break someone.

bonzini · 2019-02-22T13:06:52Z

Am I correct in reading the code that this is to adding a link argument like -Wl,-u,foofunc for all the missing functions?

Yes. Note however that -u is just a hint. Compare these two:

$ cat f.c
int main() { }
$ gcc f.c -Wl,-u,bar
$ gcc f.c -Wl,--require-defined=bar
/usr/bin/ld: required symbol `bar' not defined
collect2: error: ld returned 1 exit status

What do you suggest? A built-in option?

jpakkane · 2019-02-24T22:56:18Z

Well, AIUI this can be done without these changes with something like this:

required_symbols_args = ['-Wl,--require-defined=bar', ...]

executable(...,
     link_args: required_symbol_args)

The downside being that you don't get the list of symbols automatically. On the other hand listing them out is more explicit which some people consider to be an advantage. I guess it depends how often the list of symbols changes...

bonzini · 2019-02-25T09:54:17Z

I guess it depends how often the list of symbols changes...

It can change quite often. With our current build system it's totally transparent so I don't know how often it would change, but it would likely be a blocker for switching to Meson. We never had a compatibility problem.

I guess it would be okay to keep the logic in the build system instead of moving it to Meson. There are two issues for this:

VC++ does not support nested response files so it would not be possible to support it. But my project does not care so this is not a blocker.
You cannot order a custom_target before an executable. For example:

    project('a', 'c')
    lib_util = static_library('util', 'x.c', pic: true)
    mod_b = shared_module('b', 'y.c')
    target = custom_target('undefsym', output: ['x.undef'], input: mod_b,
                           command: ['undefsym.sh', '@INPUT@'])
    lib_util_full = declare_dependency(link_with: [lib_util], link_args: '@x.undef')
    executable('a', 'f.c', dependencies: [lib_util_full])

but there is no way for lib_util_full to add an implicit dependency from the executable to x.undef. If you can suggest how to extend declare_dependency I can work on it. Maybe with link_args: target?

jpakkane · 2019-02-26T20:05:50Z

Well one way of doing this currently, but which granted is a bit hacky is to do something like:

# UNDEF_DEFINITION START
<add here all the link args needed>
# UNDEF_DEFINITION END

run_target(# a script that takes the files as input, opens this file and updates the linker 
                  # command list above
)

This would not be automatic (you'd need to execute the run target manually when symbols change) but it would work for all backends and cases.

bonzini · 2019-02-26T20:29:23Z

The set of symbols depends on the target (e.g. Linux/Windows/Mac) and especially for libraries such as glib it's not very predictable when it changes. The same patch, backported to a stable branch, might necessitate adding a symbol or two.

I guess CI would catch mistakes but still I don't think it would be acceptable for other developers.

The only solution I see is being able to add implicit dependencies to the link rule via declare_dependency.

bonzini · 2019-02-26T20:30:05Z

(Of course the manual solution was already considered and discarded when we came up with the current scheme using nm).

bonzini · 2019-02-27T10:47:45Z

Ok, I got it to work. The trick is: 1) to build the modules in two phases using link_whole, so that the list of symbols can be gleaned from the modules' objects file, but the executable can still be linked before the shared module; 2) to use link_depends so that the symbols are updated automatically:

project('a', 'c')

modules_objs = {}
modules_objs += { 'b': static_library('b.module', 'y.c', pic: true) }
# ...

target = custom_target('undefsym', output: ['x.undef'], input: modules_objs,
                       command: [files('undefsym.sh'), '@OUTPUT0@', '@INPUT@'])

libutil = static_library('util', 'x.c', pic: true)
e = executable('a', 'f.c', link_with: [libutil], link_args: '@x.undef', link_depends: target, implib: true)
foreach name, lib : modules_objs
    shared_module(name, link_whole: lib, link_with: e)
endforeach

For the full solution, I would use get_cross_property to pass the nm binary to the script. It's not great, but I guess it's acceptable. The only extra build step compared to my patches is the ar to build libb.module.a.

jpakkane · 2019-02-28T20:38:38Z

I actually woke up last night and then immediately realized that link_depends can be used to make this work. This is not a dramatization, it really did happen. Good that you realized the same thing.

For the full solution, I would use get_cross_property to pass the nm binary to the script. It's not great, but I guess it's acceptable.

If you have set up nm in the cross file (which you should) then doing nm_exe = find_program('nm') returns the cross compilation nm rather than the native one.

elmarco · 2019-07-21T20:13:03Z

I managed to build the join of undefs & defs as a custom target, but I wonder what tricks I could use to transform the result (content of target file) as link_args. I wish I could have a custom_target().stdout() for example. (similar to run_command() run result object). Any idea?

bonzini · 2019-07-21T23:20:16Z

You can use @foo to include the content of a file into the command line (this is a gcc feature that is also supported by clang).

bonzini force-pushed the wl-u branch from e1dd57a to a451274 Compare June 4, 2018 13:53

ignatenkobrain reviewed Jun 4, 2018

View reviewed changes

bonzini force-pushed the wl-u branch 2 times, most recently from 07ad5bf to 7eb1cfa Compare June 5, 2018 12:37

ignatenkobrain reviewed Jun 5, 2018

View reviewed changes

bonzini force-pushed the wl-u branch 2 times, most recently from 82c3562 to 688f379 Compare June 5, 2018 12:59

ignatenkobrain reviewed Jun 5, 2018

View reviewed changes

mesonbuild/scripts/undefsymbols.py

# libraries into the main program.

import sys

from ..mesonlib import Popen_safe

This comment was marked as resolved.

Sign in to view

bonzini force-pushed the wl-u branch from 688f379 to 399539f Compare June 5, 2018 13:01

nirbheek reviewed Jun 5, 2018

View reviewed changes

bonzini force-pushed the wl-u branch 5 times, most recently from cf33dec to 0c3587b Compare June 11, 2018 17:28

bonzini mentioned this pull request Jun 12, 2018

File substitution ninja-build/ninja#1447

Closed

bonzini added 2 commits February 4, 2019 09:31

bonzini force-pushed the wl-u branch 3 times, most recently from d575f0a to f6bc04e Compare February 6, 2019 18:18

bonzini force-pushed the wl-u branch from f6bc04e to 96594d7 Compare February 7, 2019 12:28

compilers: support listing undefined symbols with nm

007b3c4

Finally, this patch adds support to undefsymbols for the nm utility.

bonzini force-pushed the wl-u branch from 62198e0 to 007b3c4 Compare February 7, 2019 13:39

compilers: disable undefsymbols when bitcode output is active

5e98884

Linker option -u is incompatible with -bitcode_bundle. Recognize this and avoid using the .undef files on the linker command line. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

bonzini force-pushed the wl-u branch from 8f0a810 to 5e98884 Compare February 7, 2019 14:34

bonzini closed this Feb 27, 2019

bonzini deleted the wl-u branch February 1, 2021 08:00

Identify statically-linked symbols that are needed by modules #3683

Identify statically-linked symbols that are needed by modules #3683

bonzini commented Jun 4, 2018

ignatenkobrain Jun 4, 2018

ignatenkobrain Jun 4, 2018

ignatenkobrain Jun 5, 2018

ignatenkobrain Jun 5, 2018

ignatenkobrain Jun 5, 2018

This comment was marked as resolved.

nirbheek Jun 5, 2018

bonzini Jun 5, 2018

bonzini commented Jun 15, 2018

jpakkane commented Jun 17, 2018

bonzini commented Jun 18, 2018

bonzini commented Feb 7, 2019

jpakkane commented Feb 21, 2019

bonzini commented Feb 22, 2019

jpakkane commented Feb 24, 2019

bonzini commented Feb 25, 2019 •

edited

jpakkane commented Feb 26, 2019

bonzini commented Feb 26, 2019

bonzini commented Feb 26, 2019

bonzini commented Feb 27, 2019 •

edited

jpakkane commented Feb 28, 2019

elmarco commented Jul 21, 2019

bonzini commented Jul 21, 2019

Identify statically-linked symbols that are needed by modules #3683

Identify statically-linked symbols that are needed by modules #3683

Conversation

bonzini commented Jun 4, 2018

ignatenkobrain Jun 4, 2018

Choose a reason for hiding this comment

ignatenkobrain Jun 4, 2018

Choose a reason for hiding this comment

ignatenkobrain Jun 5, 2018

Choose a reason for hiding this comment

ignatenkobrain Jun 5, 2018

Choose a reason for hiding this comment

ignatenkobrain Jun 5, 2018

Choose a reason for hiding this comment

This comment was marked as resolved.

nirbheek Jun 5, 2018

Choose a reason for hiding this comment

bonzini Jun 5, 2018

Choose a reason for hiding this comment

bonzini commented Jun 15, 2018

jpakkane commented Jun 17, 2018

bonzini commented Jun 18, 2018

bonzini commented Feb 7, 2019

jpakkane commented Feb 21, 2019

bonzini commented Feb 22, 2019

jpakkane commented Feb 24, 2019

bonzini commented Feb 25, 2019 • edited

jpakkane commented Feb 26, 2019

bonzini commented Feb 26, 2019

bonzini commented Feb 26, 2019

bonzini commented Feb 27, 2019 • edited

jpakkane commented Feb 28, 2019

elmarco commented Jul 21, 2019

bonzini commented Jul 21, 2019

bonzini commented Feb 25, 2019 •

edited

bonzini commented Feb 27, 2019 •

edited