`lld::elf::link` returns false from the second time on its called

[fwinter]

Using LLVM with JIT-compilation requires for some target architectures to call the linker several times, i.e. for AMD GPU kernels to turn an .o file into an .so file.

LLVM 15 and also the current code have the following problem:

bool lld::elf::link returns false from the second time on its called. I.e. the first call works fine, but subsequent calls fail.
Here's small reproducer:

#include "lld/Common/Driver.h"
#include "lld/Common/ErrorHandler.h"
#include "lld/Common/Memory.h"
#include <iostream>

int main()
{
  std::cout << "First call ... ";
  bool call_a = lld::elf::link( { "ld.lld" , "-shared" , "../kernel.o" , "-o" , "../kernel.so" } , llvm::outs(), llvm::errs(), true, true);
  std::cout << call_a << std::endl;


  std::cout << "Second call ... ";
  bool call_b = lld::elf::link( { "ld.lld" , "-shared" , "../kernel.o" , "-o" , "../kernel.so" } , llvm::outs(), llvm::errs(), true, true);
  std::cout << call_b << std::endl;
}

I tested various systems but the output is always:

First call ... 1
Second call ... 0

This behavior is agnostic to the architecture. Initially I ran into this with AMDGPU objects, but could reproduce with X86 objects.

A quick gdb session reveals that during the second call the link procedure is aborted here:

./llvm-project/lld/ELF/Driver.cpp:2624

  // Return if there were name resolution errors.
  if (errorCount())
    return;

Please help.

[llvmbot]

@llvm/issue-subscribers-lld-elf

[bjoo]

I have done some debugging here. The source of the second failure appears to be that the SymbolTable (symtab) is not cleared between the linker invocations in the test. Then on the second invocation there can be issues when inserting existing symbols, or if the test is modified to link modules that do not have common symbols, the issue can arise when the Writerin the finalizeSections() attempts to add the _DYNAMIC symbol. The second time addition fails the !isDefined() test during the Symbol::resolve() method causing the rest of the original symbol not to be copied in (Symbol::shouldReplace()) fails. At this point the STB_WEAK binding is not copied into the new symbol table entry, amongst other things, leading to an Assertion failure later on.

I did not find a suitable way of resetting the symbol table, looking at the LLVM sources, however, I did see a cleanup function defined in the Driver.cpp which looks like:

 ctx->e.cleanupCallback = []() {
    elf::ctx.reset();
    symtab = SymbolTable();

    outputSections.clear();
    symAux.clear();

    tar = nullptr;
    in.reset();

    partitions.clear();
    partitions.emplace_back();

    SharedFile::vernauxNum = 0;
  };

Using this one can try to cleanup at the start of each link simply by calling:

ctx->e.cleanupCallback();

immediately after defining it. However this sets the symbol table as

symtab = SymbolTable();

and I do not know if that cleanly frees a previously defined symbol table. (My initial suspicion is maybe not..). So a cleaner solution would likely be better.

[aganea]

@fwinter Can you try using lld::safeLldMain instead of lld::elf::link? (and build lld/tools/lld/lld.cpp as part of your executable). After https://reviews.llvm.org/D119049 that API should be easier to use I hope. D119049 also adds some unittests similar to your usage.

[fwinter]

Hi @aganea
Could solve the issue by mimicking what safeLldMain is doing. Great, thanks!