[AMDGPU][LLD][Clang] Passing arguments via -mllvm to pass plugin causes unknown argument in lld

[Thyre]

Description

I'm in the process of writing a pass plugin for instrumentation of applications compiled with LLVM. While checking for function names which need to be filtered, I tried to compile a bit of CUDA and HIP code. During this, I encountered an issue when trying to pass an argument to the pass plugin.

Right now, I'm passing arguments to the pass plugin via -mllvm [arg]. I found the suggestion here and it worked fine for basic code.

As far as I've seen, this is the only way to process arguments outside of using opt, right? Using opt would be extremely inconvenient, especially for projects with a lot of source files and dependencies.

Error output

For this example, I'm using the Bye example in the LLVM repository. I've removed all of the legacy registration, which should not affect the functionality.

$ clang++ -fPIC -shared Bye.cpp -o Bye.so 
$ cat min.c
int main(void){}
$ clang -x hip -fplugin=./Bye.so -O3 -fpass-plugin=./Bye.so -mllvm -wave-goodbye  minimal.c 
lld: error: -plugin-opt=-: lld: Unknown command line argument '-wave-goodbye'.  Try: '/opt/software/software/LLVM/16.0.6/bin/lld --help'
lld: Did you mean '--code-model'?
clang-16: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)

Looking at the output with clang -v, we can see that the argument is passed to lld

"/opt/software/software/LLVM/16.0.6/bin/lld" -flavor gnu -m elf64_amdgpu --no-undefined -shared -plugin-opt=-amdgpu-internalize-symbols -plugin-opt=mcpu=gfx906 -plugin-opt=-wave-goodbye -o /tmp/minimal-6ad3b7/minimal-gfx906.out /tmp/minimal-f23b1b/minimal-gfx906.o

Trying to compile the same tool with -x cuda works fine.

$ clang -fuse-ld=/opt/software/software/LLVM/16.0.6/bin/ld.lld -x cuda --cuda-gpu-arch=sm_75 -fplugin=./Bye.so -fpass-plugin=./Bye.so -mllvm -wave-goodbye minimal.c -O3
clang-16: warning: CUDA version is newer than the latest partially supported version 11.8 [-Wunknown-cuda-version]
Bye: main

The main question here is: Is there a way to avoid passing the parameter to lld when working with clang -x hip (or hipcc with ROCm)?

[yxsamliu]

you can use -Xarch_host -mllvm=-wave-goodbye, then the -mllvm option will not be passed to lld

[jhuber6]

you can use -Xarch_host -mllvm=-wave-goodbye, then the -mllvm option will not be passed to lld

Are we passing -mllvm options to the linker by default? That doesn't seem right, the regular clang driver only does that with -Wl,-mllvm IIRC.

[yxsamliu]

that's a good point. we should stop passing -mllvm options to device lld. Since we use -Xoffload-linker instead of -Wl to pass options to device lld, we should use -Xoffload-linker -mllvm= to pass -mllvm to device linker.

[yxsamliu]

fix by https://reviews.llvm.org/D154145

[llvmbot]

@llvm/issue-subscribers-clang-driver

[Thyre]

you can use -Xarch_host -mllvm=-wave-goodbye, then the -mllvm option will not be passed to lld

Thanks for the tip. I can confirm that the code can compile when using -Xarch_host. However, with LLVM 15 / LLVM 16 passed arguments are ignored when the plugin runs. I've also tested this with aomp 17.0-0 and LLVM 17.0.0-git. Here, the arguments are passed to the plugin and no warning occurs.

LLVM:

$ ml LLVM                                                                                                                                                                                       
$ clang --version                                                                                                                                                                              
clang version 16.0.6 (git@github.com:llvm/llvm-project.git 7cbf1a2591520c2491aa35339f227775f4d3adf6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/software/software/LLVM/16.0.6/bin
$ clang -fPIC -shared bye.cpp -o bye.so -fno-rtti                                                                                                                                              
$ clang -fplugin=./bye.so -fpass-plugin=./bye.so -O3 minimal.c -Xarch_host -mllvm=-wave-goodbye                                                                                                 
clang-16: warning: argument unused during compilation: '-Xarch_host -mllvm=-wave-goodbye' [-Wunused-command-line-argument]

aomp 17.0-0:

$ ml ROCm                                                                                                                                                                                      
$ clang --version                                                                                                                                                                               
AOMP_STANDALONE_17.0-0 clang version 17.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/software/software/ROCm/aomp_17.0/bin
$ clang -fPIC -shared bye.cpp -o bye.so -fno-rtti                                                                                                                                         
$ clang -fplugin=./bye.so -fpass-plugin=./bye.so -O3 minimal.c -Xarch_host -mllvm=-wave-goodbye                                                                                                 
Bye: main

[yxsamliu]

the older compiler does not support -mllvm=, which is added recently