Support LTO linker plugin

[marxin]

Similarly to #93, please consider adding -flto support for GCC bytecode.
Your linker looks very promising and LTO is getting more commonly used. Note there are distros like openSUSE, Fedora, (Ubuntu, Debian in near future), that use LTO by default for package builds.
Thanks.

[rui314]

Yes, I want to do that. My plan is to support the standard linker plugin API, so that mold works both LLVM and GCC. I just don't have enough time to do that at the moment.

[MaskRay]

I think it is difficult to have a plugin to support both LLVM and GCC. plugin-api.h style ld_plugin_symbol is very different from LLVM llvm::lto::SymbolResolution.

[rui314]

If so, how did it work with GNU ld and gold?

[janhubicka]

Both GCC and LLVM support the linker API (you need to build LLVM with buinultils include path to get the gold plugin). What would be really nice (and technically possible I think) would be to support multiple plugins at once. So if you have some object files bullt with GCC and other with LLVM you get LTO at least within the partitions of program built with the same compiler.

[rui314]

I tried to write code to support the LTO linker plugin only to find that the API is not suitable for mold. The linker plugin API is designed with the conventional linker design in mind, and mold is quite different from that.

Here is an example: the plugin API provide a function to ask if the plugin wants to "claim" the ownership of an object file. That function claims the ownership of an object file if the file was compiled with -flto. If the plugin takes an ownership of a file, it reads a symbol table of the file and calls a callback to notify to the linker the contents of the symbol table. Finally, the plugin compiles all files it owns and pass the result back to the linker.

So, in this API, there's no way to read a symbol table without including the file into the final LTO result. This API design doesn't work for us because mold reads object files from archive files even if they won't be included into the final result.

It looks like this blocks us from using the plugin API. We probably need to propose a new API to both GCC and LLVM. That's I guess not easy. My hope is that there might be a chance to get a favorable treatment by GCC as GNU doesn't have a faster linker anymore (gold is quasi-deprecated, and lld is unlikely to support the linker plugin API because it instead directly uses LLVM), so they might have an incentive on their side to make their compiler work well with mold.

[janhubicka]

Rui, thank you for looking into this. The linker plugin API is extensible (it has versions and it was extended in the past), So if needed, we can discuss a new version of the plugin API. I maintain the GCC LTO implementation and would be happy to help with this. Gold maintainers are Iant Taylor and Cary Countant. For BFD ld implementation I think H.J. Lu is good contact. The reason mold reads all files in archive right away is to achieve parallelism of linking process and later it discards symbols that are not needed? One need to keep in mind that LTO linking is relatively slow in the end (because all compilation needs to be done) so I am not sure how much benefits the parallelism gives. But I wonder if one can simply use multiple instances of the plugin - in the "pre-linking" stage one can invoke the plugin and check if a file is accepted by it and to obtain symbol tables. Similar usage of linker is done by nm, ar and other tools. Once the final set of files is determined feed the files again into a new instance of the plugin plugin and do the final job. lto-plugin in GCC is quite cheap wrapper that does 1) determine if file is GCC LTO file 2) reads the symbol table and passes it to the linker 3) once pre-linking is done it stores the symbol table given by linker to a resolution file 4) it invokes lto-wrapper which eventually invokes GCC in a separate process. There is relatively little bookkeeping between 2) and 3) so I do not see it as much of a complication if some files got discarded between these two steps. Adding an extra LDPT method for discarding previously claimed files is IMO quite easy (if agreed on with the binutils ld&gold maintainers) and it would save some effort in duplicated loading of symbol tables. Perhaps it would also make it possible to make linker plugin thread safe to some extend to support faster prelinking. Honza

…

On Tue, Jan 4, 2022 at 11:58 AM Rui Ueyama ***@***.***> wrote: I wrote code to support the LTO linker plugin only to find that the API is not suitable for mold. The linker plugin API is designed with the conventional linker design in mind, and mold is quite different from that. Here is an example: the plugin API provide a function to ask if the plugin wants to "claim" the ownership of an object file. That function claims the ownership of an object file if the file was compiled with -flto. If the plugin takes an ownership of a file, it reads a symbol table of the file and calls a callback to notify to the linker the contents of the symbol table. Finally, the plugin compiles all files it owns and pass the result back to the linker. So, in this API, there's no way to read a symbol table without including the file into the final LTO result. This API design doesn't work for us because mold reads object files from archive files even if they won't be included into the final result. It looks like this blocks us from using the plugin API. We probably need to propose a new API to both GCC and LLVM. That's I guess not easy. My hope is that there might be a chance to get a favorable treatment by GCC as GNU doesn't have a faster linker anymore (gold is quasi-deprecated, and lld is unlikely to support the linker plugin API and instead directly uses LLVM), so they might have an incentive on their side to make their compiler to work well with mold. — Reply to this email directly, view it on GitHub <#181 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK7OQW2DMLUGWXXFN6MTVPTUULHEVANCNFSM5KSCFIAA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[rui314]

Hi Honza,

Thank you for your comment! It was nice to have you here to discuss how to support GCC LTO support in mold.

We do read archive members right away for parallelism. We parse all input files (including the ones in archive files) simultaneously at the beginning of the process, so that the symbol resolution stage doesn't have to read any files from the disk. I understand this speedup doesn't matter much when LTO is in use, but this is how mold resolves symbols, and I don't think we want to implement a different algorithm only for LTO (not only to save the amount of code but also to guarantee that symbols are resolved in the exact same way both in a non-LTO and LTO builds).

So, yes, what I immediately needed is an API to discard previously claimed files.

But can we add a convenient API to read symbols from an LTO object file without claiming it instead? That would be more convenient than letting the plugin claim a file and later letting it forget about it.

[janhubicka]

On the GCC side I think adding an extra API hook reading symbol tables and not registering it should be easy. See claim_file_handler in lto-plugin.c. It essentially checks the file, then it calls add_symbols or add_symbols_v2 and then it adds the file to the list of claimed files. You essentially want to split this into two API calls - check_file_and_read_symbol_table and reigster_file it seems.

…

On Tue, Jan 4, 2022 at 2:25 PM Rui Ueyama ***@***.***> wrote: Hi Honza, Thank you for your comment! It was nice to have you here to discuss how to support GCC LTO support in mold. We do read archive members right away for parallelism. We parse all input files (including the ones in archive files) simultaneously at the beginning of the process, so that the symbol resolution stage doesn't have to read any files from the disk. I understand this speedup doesn't matter much when LTO is in use, but this is how mold resolves symbols, and I don't think we want to implement a different algorithm only for LTO (not only to save the amount of code but also to guarantee that symbols are resolved in the exact same way both in a non-LTO and LTO builds). So, yes, what I immediately needed is an API to discard previously claimed files. But can we add a convenient API to read symbols from an LTO object file without claiming it instead? That would be more convenient than let the plugin claim a file and later let it forget about it. — Reply to this email directly, view it on GitHub <#181 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK7OQW7EIAWONBFYTVREUMDUULYLDANCNFSM5KSCFIAA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[marxin]

But can we add a convenient API to read symbols from an LTO object file without claiming it instead? That would be more convenient than letting the plugin claim a file and later letting it forget about it.

Hey. I've just implemented the hook ld_plugin_open_and_read_symbols_handler which returns true if an object contains LTO symbols. It's likely not going to work with LTO offloading, but we can take a look at the later:
https://gcc.gnu.org/wiki/Offloading

So please let's use the following GCC branch of testing and development:
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/devel/mold-lto-plugin

The branch will be pulled by default if you clone the repo.

@rui314 Can you please test it and provide feedback?

[janhubicka]

Martin, thanks for implementing this! It may be enough to get plugin API working for mold. For effectivity we will however eventually want ld_plugin_register_file to bypass the symtab reading which is done by claim_file_handler and only adds the filename to the list of objects to pass to lto-wrapper.

…

On Tue, Jan 4, 2022 at 4:42 PM Martin Liška ***@***.***> wrote: But can we add a convenient API to read symbols from an LTO object file without claiming it instead? That would be more convenient than letting the plugin claim a file and later letting it forget about it. Hey. I've just implemented the hook ld_plugin_open_and_read_symbols_handler which returns true if an object contains LTO objects. It's likely not going to work with LTO offloading, but we can take a look at the later: https://gcc.gnu.org/wiki/Offloading So please let's use the following GCC branch of testing and development: https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/devel/mold-lto-plugin The branch will be pulled by default if you clone the repo. @rui314 <https://github.com/rui314> Can you please test it and provide feedback? — Reply to this email directly, view it on GitHub <#181 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK7OQW6NSOMA2GYLCW75XDLUUMINNANCNFSM5KSCFIAA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[rui314]

@marxin Let me experiment with your change. I'll get back to you in a few days.

[marxin]

For effectivity we will however eventually want ld_plugin_register_file to bypass the symtab reading which is done by claim_file_handler and only adds the filename to the list of objects to pass to lto-wrapper.

Sure, I'm planning to do that. It will be simple.

[joshcangit]

Looking forward to when mold will support LTO.

I was thinking maybe use gold as the LTO fallback for gcc but maybe bfd is better for compatibility.

I just realized it's ld.bfd not bfd.
It's there in Ubuntu and Termux just that the symlink is missing on Termux.

[rui314]

@joshcangit GNU ld is still the de-facto standard linker on Linux, so it is very unlikely that you don't have it but gold and lld on your system. What distro are you using? What ld --version says on your system?

[marxin]

@rui314 Hey. Have you find a time experimenting with GCC LTO plug-in support?

[rui314]

@marxin Not yet. I'm receiving more bug reports than I expected and I haven't had a chance to work on LTO yet.

[rui314]

@marxin I investigated more about the LTO linker plugin interface and found that our usage is actually supported by the standard API. The point is that even though we have to let the linker plugin to claim the ownership of an LTO object file to read its symbol table, we can later let it ignore any file by returning LDPT_NO_SYMS as a response from the get_symbols callback.

So it looks like we can do LTO with the existing API, which is a good news because we can support old versions of GCC and LLVM.

[rui314]

It's still work-in-progress, but with the above change, mold can now link itself with -flto. It works both with GCC and Clang.

[marxin]

These are all great news. I'm going to be off next week, so I'll start with intensive testing the week after that.

[rui314]

It looks like I can now build clang with LTO using clang and mold. But I can't do the same build using gcc and mold. It's because the GCC linker plugin is still using get_symbols_v2 API instead of get_symbols_v3 API, so there's no way to tell to the plugin that an archive member wasn't chosen to be included in the final linker output.

[marxin]

Looking at the plug-in, do I understand it correctly that what we need is to add get_symbols_v3 to GCC's LTO plug-in and skip resolution info emission if LDPT_NO_SYMS is returned by the linker. Am I correct?

[rui314]

Correct. I don't know if it will fix all the remaining problems, but that v3 API is clearly missing in the GCC
LTO plugin. It is odd that GCC defined only the enums for the v3 API and didn't implement the feature.

[satmandu]

As per xbmc/xbmc#20891 (comment) it appears that flto=thin is not working with mold.

[roland-5]

I build mold with commit 4caadef now and could use, after this -Wl,--thinlto-jobs=all (LLVM/CLANG) to builds apps without problem. THANK YOU!

[marxin]

About the detection of if get_symbols_v3 is used. I'm suggesting the following extension to the plug-in API that provides a name and compiler version of a linker. Note it's very similar to LDPT_GOLD_VERSION = 2, LDPT_GNU_LD_VERSION.
The current implementation returns GCC 12.0.1.

I'm planning suggesting the patch in next GCC stage1 (after 12.1 gets released):
0001-lto-plugin-add-LDPT_PLUGIN_VERSION.patch.txt

@rui314 What do you think about it?

[rui314]

I think that the string-based version detection is a bit fragile and can cause chaos just like web browser's User-Agent string. I could imagine that other compiler would have to return something like "GCC 12.0.1 (LLVM 20.0)" to make it compatible with older linker plugins.

Maybe a silly question, but why can't we just define a symbol other than onload? What we want is to test whether a plugin supports v3 API or not, so checking the existence of some particular symbol (e.g. dlsym(handler, "supports_v3_api") == nullptr) should satisfy our need. (Which brings us to a bigger question -- why is everything in the plugin API is organized as a callback? It's frankly just hard to use.)

[marxin]

I think that the string-based version detection is a bit fragile and can cause chaos just like web browser's User-Agent string. I could imagine that other compiler would have to return something like "GCC 12.0.1 (LLVM 20.0)" to make it compatible with older linker plugins.

So do you prefer a numeric version similar to GCC_VERSION macro? The version is the identification of the compiler for which the plug-in is done, so GCC 12.0.1 (LLVM 20.0) doesn't make sense.

Maybe a silly question, but why can't we just define a symbol other than onload? What we want is to test whether a plugin supports v3 API or not, so checking the existence of some particular symbol (e.g. dlsym(handler, "supports_v3_api") == nullptr) should satisfy our need.

I would like to come up with a more generic mechanism and not a special case for support_get_symbols_v3. One can deduce functionality based on version, similarly to what various configure checks do.

(Which brings us to a bigger question -- why is everything in the plugin API is organized as a callback? It's frankly just hard to use.)

Good question, maybe @janhubicka can answer ;)

[rui314]

So do you prefer a numeric version similar to GCC_VERSION macro? The version is the identification of the compiler for which the plug-in is done, so GCC 12.0.1 (LLVM 20.0) doesn't make sense.

I don't want to identify the whole compiler's version but each feature's version. I want to know whether or not a given linker plugin supports version N of feature X, and we are not really interested in what the compiler version is.

Identifying a compiler version number instead of each feature's version has a few problems:

1. The linker has to manage which compiler version supports what feature set. Keeping that in sync with newer compilers is laborsome.
2. Hypothetically, assume that I create a new C++ compiler which does support LTO. My compiler's plugin can't return "MyNewC++Compiler 1.0" to the linker because no one would recognize that string because it's new and thus all linkers would fall back to the most basic feature set when using my compiler's linker plugin. So I probably would have to return "GCC 12.0.1" (or "GCC 12.0.1 (MyNewC++Compiler 1.0)") instead to force the linker to use new feature set. That's what I said as the web browser's User-Agent problem.

[marxin]

I don't want to identify the whole compiler's version but each feature's version. I want to know whether or not a given linker plugin supports version N of feature X, and we are not really interested in what the compiler version is.

Yes, I understand that. That's pretty reasonable requirement, let me think what we can do about it.

[rui314]

The other thing I'd like to identify is whether I need to close an file descriptor after claim_file_hook or not. It looks like GCC doesn't take the ownership of a given fd while LLVM does, so we close an fd only for GCC to avoid the "too many open files" issue.

The code to close an fd is here: https://github.com/rui314/mold/blob/main/elf/lto.cc#L542

[marxin]

Looking at both bfd and gold, they are really responsible for fd closure:

bfd:

static int
try_claim (bfd *abfd)
{
  int claimed = 0;
  struct ld_plugin_input_file file;

  file.handle = abfd;
  if (bfd_plugin_open_input (abfd, &file)
      && current_plugin->claim_file)
    {
      current_plugin->claim_file (&file, &claimed);
      bfd_plugin_close_file_descriptor ((abfd->my_archive != NULL
					 ? abfd : NULL),
					file.fd);
    }

  return claimed;
}

gold:

void
Plugin_manager::cleanup()
{
  if (this->any_added_)
    {
      // If any input files were added, close all the input files.
      // This is because the plugin may want to remove them, and on
      // Windows you are not allowed to remove an open file.
      close_all_descriptors();
    }

So there's likely a difference in between GCC and LLVM. Is it problematic for mold? Your code seems reasonable to me.

[Alcaro]

Yes, it is problematic for mold. There is a workaround, but it's pretty ugly. #362

[marxin]

There is a workaround, but it's pretty ugly. #362

Why is it ugly?
Apparently, the GCC LTO plugin implementation (which is older), does not close the descriptors and the behavior was likely never specified. So it's now implementation-defined and we should preserve it for existing linkers like BFD and GOLD.

[rui314]

It's ugly because it uses the plugin file name to decide whether we should close the fd or not. If a plugin name ends with LLVMgold.so, we don't close the fd because it's LLVM. Otherwise, we close it because it's GCC. But obviously this logic is fragile (there's no guarantee that LLVM always uses that filename for their plugin.)

[marxin]

Well, that brings us back to the suggested tv_plugin_version callback. I know the situation is not ideal, but such detection seems to be reasonable. You can detect the GCC plugin by name and consider others as LLVM plugin. To be honest, I don't expect any new plugin implementation in a near future.

[rui314]

Yeah, right. I still don't like if (LLVM) or if (GCC)-style dispatching, but that's probably practical. Given this situation, I'm OK with your proposal of adding LDPT_PLUGIN_VERSION to the plugin API.

[marxin]

Yeah, right. I still don't like if (LLVM) or if (GCC)-style dispatching, but that's probably practical. Given this situation, I'm OK with your proposal of adding LDPT_PLUGIN_VERSION to the plugin API.

Fine, I'll do that once GCC's stage1 opens.

About the undefined symbols problem. I've just tried the current master and I'm able to build e.g. postgresql that was affected by the problem. Have you made mold somehow more robust about it? Or am I just lucky?

[rui314]

I made a change to mold so that after the plugin returns an ELF file to us, we redo symbol resolution from scratch instead of trying to replace IR symbols with the ELF symbols. I think it makes the mold's symbol resolution more robust for LTO. mold can now pull out object files from archives if they are needed as a result of LTO.

[marxin]

I made a change to mold so that after the plugin returns an ELF file to us, we redo symbol resolution from scratch instead of trying to replace IR symbols with the ELF symbols. I think it makes the mold's symbol resolution more robust for LTO. mold can now pull out object files from archives if they are needed as a result of LTO.

Oh great!

One more thing: can you please add auto-loading support for plugins, similarly to what binutils does:

... based plugin it must first
be copied into the ‘${libdir}/bfd-plugins’ directory. All gcc based linker
plugins are backward compatible, so it is sufficient to just copy in the newest
one.

?

[rui314]

For the sake of build reproducibility and cross compilation, we depend only on command line arguments. That is, if you pass the exact same command line arguments and their file contents are the same, we promise we behave exactly the same. For this reason, mold (as well as lld) does not have a notion of implicit default library path or such. All target-dependent knowledge are coded to the compiler and passed from the compiler to the linker. It looks like this policy worked really well, so I don't want to hard-code the plug-in directory to the linker. Can you always pass the latest version of the plugin path to the linker instead?

[marxin]

Oh, you are correct. The autoloading mechanism is actually designed for tools like nm, ranlib or ar.
I've just pushed 2 commits to master that enable plug-in when one configures GCC with --with-ld=which ld.mold:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c083e654bd0f29a365ec957c4c0d4e713fb0b010

[marxin]

There's v2 of the get_version API. I decided for use of an enum and integer version (similar to gold_version).
I'm going to send the patch in next stage1:
0001-LTO-plugin-add-ld_plugin_version-callback.patch.txt

[rui314]

Thanks. Your patches look good to me.

[marxin]

Let's close this, the full support will be available since GCC 13.1.

[moncefmechri]

I'm a bit confused. My understanding from this thread is that GCC+LTO+mold work together starting from GCC 13.1.

But I tried a toy example with GCC 12.2 and mold 2.4.0 and everything seems to work.

$ cat foo.h
#pragma once

void foo();

$ cat foo.c
#include "foo.h"

#include <stdio.h>

void foo()
{
    printf("foo\n");
}

$ cat bar.h 
#pragma once

void bar();

$ cat bar.c
#include "foo.h"

#include <stdio.h>

void bar()
{
    printf("bar\n");
}

$ cat main.c
#include "foo.h"
#include "bar.h"

int main()
{
    foo();
    bar();
}

$ gcc -flto -c foo.c
$ gcc -flto -c bar.c
$ gcc -flto -c main.c
$ gcc  -fuse-ld=mold -o main main.o foo.o bar.o
$ ./main
foo
bar

I also tried an older mold that I had lying around (1.10.1) and the same command above succeeded too.

Finally, I tried to use lld instead of mold, and, as expected, the build now fails given that lld does not support GCC LTO
What am I missing?

[rui314]

mold can do LTO for older versions of GCC, but that was implemented with a bit of hack (the linker restarts itself during LTO). The hack is not necessary for the recent versions of GCC. Anyway, it's not visible from user, so you don't need to worry about it.

[raininja]

If I have the following error, is it the same general error as this plugin? I was sure to build the offending object with the same GCC . . .

mold: fatal: LocalStorageProtocol.pb-c.c.o: not claimed by the LTO plugin; please make sure you are using the same compiler of the same version for all object files

[rui314]

@raininja I'm almost certain that the error message was correct. How did you verify that you are using the same version of a compiler to compile that object file and to invoke the linker?