[AArch64][PGO] Deadlock in PGO value profiling on AArch64

[gulfemsavrun]

We ran into this issue during a multi-stage build where we built our toolchain with PGO. The hang happens specifically after we build Clang with PGO enabled, and then use that PGO-optimized Clang to build the compiler-rt. The linker stalls when compiling some runtime tests like:

llvm-project/compiler-rt/cmake/config-ix.cmake

Line 98 in 597f93d

check_cxx_compiler_flag(-fomit-frame-pointer COMPILER_RT_HAS_FOMIT_FRAME_POINTER_FLAG)

I attached gdb to the stalled lld process and found the following call stack:

#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S:50
#1  0x0000ffffa4a22658 in __internal_syscall_cancel (a1=a1@entry=281474580353104, a2=a2@entry=393, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=nr@entry=98) at ./nptl/cancellation.c:49
#2  0x0000ffffa4a229f0 in __futex_abstimed_wait_common64 (private=0, futex_word=0xffffe8601050, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#3  __futex_abstimed_wait_common (futex_word=0xffffe8601050, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#4  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xffffe8601050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at ./nptl/futex-internal.c:139
#5  0x0000ffffa4a253ec in __pthread_cond_wait_common (cond=0xffffe8601030, mutex=0xffffe8601000, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:426
#6  ___pthread_cond_wait (cond=0xffffe8601030, mutex=0xffffe8601000) at ./nptl/pthread_cond_wait.c:458
#7  0x0000aaaaed0f8ea4 in std::__2::__libcpp_condvar_wait[abi:ne220000](pthread_cond_t*, pthread_mutex_t*) (__cv=0xffffe8601050, __m=0x189)
    at /home/gulfem/llvm-project-gulfem/libcxx/include/__thread/support/pthread.h:122
#8  std::__2::condition_variable::wait (this=0xffffe8601050, lk=...) at ../../../llvm-project-gulfem/libcxx/src/condition_variable.cpp:36
#9  0x0000aaaaecfcfa3c in std::__2::condition_variable::wait[abi:nn220000]<llvm::parallel::detail::Latch::sync() const::{lambda()#1}>(std::__2::unique_lock<std::__2::mutex>&, llvm::parallel::detail::Latch::sync() const::{lambda()#1}) (this=0xffffe8601030, __lk=..., __pred=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__condition_variable/condition_variable.h:112
#10 llvm::parallel::detail::Latch::sync (this=0xffffe8600ff8) at llvm/include/llvm/Support/Parallel.h:85
#11 llvm::parallel::TaskGroup::~TaskGroup (this=0xffffe8600ff8) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:191
#12 0x0000aaaae45d6b50 in (anonymous namespace)::Writer<llvm::object::ELFType<(llvm::endianness)1, true> >::writeSections (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:3026
#13 (anonymous namespace)::Writer<llvm::object::ELFType<(llvm::endianness)1, true> >::run (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:376
#14 lld::elf::writeResult<llvm::object::ELFType<(llvm::endianness)1, true> > (ctx=...) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:100
#15 0x0000aaaae42f1c6c in lld::elf::LinkerDriver::link<llvm::object::ELFType<(llvm::endianness)1, true> > (this=<optimized out>, this@entry=0xaaab2a1035c0, args=...)
    at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:3503
#16 0x0000aaaae42ce1f4 in lld::elf::LinkerDriver::linkerMain (this=<optimized out>, this@entry=0xaaab2a1035c0, argsArr=...) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:729
#17 0x0000aaaae42caba4 in lld::elf::link (args=..., stdoutOS=..., stderrOS=..., exitEarly=<optimized out>, disableOutput=<optimized out>) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:140
#18 0x0000aaaae418d010 in lld::unsafeLldMain (args=..., stdoutOS=..., stderrOS=..., drivers=..., exitEarly=true) at ../../../../llvm-project-gulfem/lld/Common/DriverDispatcher.cpp:163
#19 0x0000aaaae3bcd3d0 in lld_main (argc=argc@entry=32, argv=argv@entry=0xffffe8602f38) at ../../../../llvm-project-gulfem/lld/tools/lld/lld.cpp:90
#20 0x0000aaaae418af4c in findTool (Argc=32, Argv=0xffffe8602f38, Argv0=0xffffe8603e8c "/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/ld.lld")
    at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:68
#21 0x0000aaaae4185830 in main (Argc=32, Argv=0xffffe8602f38) at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:85

I can sometimes reproduce the hang when running tests like the ones mentioned, but the issue is not consistent:

[1/2] /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot -DCOMPILER_RT_HAS_Z_TEXT  --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -MD -MT CMakeFiles/cmTC_f3b74.dir/src.c.obj -MF CMakeFiles/cmTC_f3b74.dir/src.c.obj.d -o CMakeFiles/cmTC_f3b74.dir/src.c.obj -c /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/CMakeFiles/CMakeScratch/TryCompile-gy54Ab/src.c
clang: warning: -Wl,-z,text: 'linker' input unused [-Wunused-command-line-argument]
[2/2] : && /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -L/home/gulfem/fuchsia-idk/arch/x64/lib -fuse-ld=lld CMakeFiles/cmTC_f3b74.dir/src.c.obj -o cmTC_f3b74  -lc && :

This is the source file for these tests:

int main(void) { return 0; }

This bug is specific to PGO builds on AArch64; it does not occur on x86 architectures or in non-PGO builds.

The hang started happening after we adopted IR PGO instead of the Front-End PGO via #156060.

[llvmbot]

@llvm/issue-subscribers-lld-elf

Author: None (gulfemsavrun)

We ran into this issue during a multi-stage build where we built our toolchain with PGO. The hang happens specifically after we build Clang with PGO enabled, and then use that PGO-optimized Clang to build the compiler-rt. The linker stalls when compiling some runtime tests like: https://github.com/llvm/llvm-project/blob/597f93d36b035faeb63f4ba0d61a8b8e25eddaab/compiler-rt/cmake/config-ix.cmake#L98

I attached gdb to the stalled lld process and found the following call stack:

#<!-- -->0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S:50
#<!-- -->1  0x0000ffffa4a22658 in __internal_syscall_cancel (a1=a1@<!-- -->entry=281474580353104, a2=a2@<!-- -->entry=393, a3=a3@<!-- -->entry=0, a4=a4@<!-- -->entry=0, a5=a5@<!-- -->entry=0, a6=a6@<!-- -->entry=4294967295, nr=nr@<!-- -->entry=98) at ./nptl/cancellation.c:49
#<!-- -->2  0x0000ffffa4a229f0 in __futex_abstimed_wait_common64 (private=0, futex_word=0xffffe8601050, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#<!-- -->3  __futex_abstimed_wait_common (futex_word=0xffffe8601050, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#<!-- -->4  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@<!-- -->entry=0xffffe8601050, expected=expected@<!-- -->entry=0, clockid=clockid@<!-- -->entry=0, abstime=abstime@<!-- -->entry=0x0, private=private@<!-- -->entry=0)
    at ./nptl/futex-internal.c:139
#<!-- -->5  0x0000ffffa4a253ec in __pthread_cond_wait_common (cond=0xffffe8601030, mutex=0xffffe8601000, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:426
#<!-- -->6  ___pthread_cond_wait (cond=0xffffe8601030, mutex=0xffffe8601000) at ./nptl/pthread_cond_wait.c:458
#<!-- -->7  0x0000aaaaed0f8ea4 in std::__2::__libcpp_condvar_wait[abi:ne220000](pthread_cond_t*, pthread_mutex_t*) (__cv=0xffffe8601050, __m=0x189)
    at /home/gulfem/llvm-project-gulfem/libcxx/include/__thread/support/pthread.h:122
#<!-- -->8  std::__2::condition_variable::wait (this=0xffffe8601050, lk=...) at ../../../llvm-project-gulfem/libcxx/src/condition_variable.cpp:36
#<!-- -->9  0x0000aaaaecfcfa3c in std::__2::condition_variable::wait[abi:nn220000]&lt;llvm::parallel::detail::Latch::sync() const::{lambda()#<!-- -->1}&gt;(std::__2::unique_lock&lt;std::__2::mutex&gt;&amp;, llvm::parallel::detail::Latch::sync() const::{lambda()#<!-- -->1}) (this=0xffffe8601030, __lk=..., __pred=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__condition_variable/condition_variable.h:112
#<!-- -->10 llvm::parallel::detail::Latch::sync (this=0xffffe8600ff8) at llvm/include/llvm/Support/Parallel.h:85
#<!-- -->11 llvm::parallel::TaskGroup::~TaskGroup (this=0xffffe8600ff8) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:191
#<!-- -->12 0x0000aaaae45d6b50 in (anonymous namespace)::Writer&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt;::writeSections (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:3026
#<!-- -->13 (anonymous namespace)::Writer&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt;::run (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:376
#<!-- -->14 lld::elf::writeResult&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt; (ctx=...) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:100
#<!-- -->15 0x0000aaaae42f1c6c in lld::elf::LinkerDriver::link&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt; (this=&lt;optimized out&gt;, this@<!-- -->entry=0xaaab2a1035c0, args=...)
    at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:3503
#<!-- -->16 0x0000aaaae42ce1f4 in lld::elf::LinkerDriver::linkerMain (this=&lt;optimized out&gt;, this@<!-- -->entry=0xaaab2a1035c0, argsArr=...) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:729
#<!-- -->17 0x0000aaaae42caba4 in lld::elf::link (args=..., stdoutOS=..., stderrOS=..., exitEarly=&lt;optimized out&gt;, disableOutput=&lt;optimized out&gt;) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:140
#<!-- -->18 0x0000aaaae418d010 in lld::unsafeLldMain (args=..., stdoutOS=..., stderrOS=..., drivers=..., exitEarly=true) at ../../../../llvm-project-gulfem/lld/Common/DriverDispatcher.cpp:163
#<!-- -->19 0x0000aaaae3bcd3d0 in lld_main (argc=argc@<!-- -->entry=32, argv=argv@<!-- -->entry=0xffffe8602f38) at ../../../../llvm-project-gulfem/lld/tools/lld/lld.cpp:90
#<!-- -->20 0x0000aaaae418af4c in findTool (Argc=32, Argv=0xffffe8602f38, Argv0=0xffffe8603e8c "/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/ld.lld")
    at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:68
#<!-- -->21 0x0000aaaae4185830 in main (Argc=32, Argv=0xffffe8602f38) at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:85

I can sometimes reproduce the hang when running tests like the ones mentioned, but the issue is not consistent:

[1/2] /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot -DCOMPILER_RT_HAS_Z_TEXT  --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -MD -MT CMakeFiles/cmTC_f3b74.dir/src.c.obj -MF CMakeFiles/cmTC_f3b74.dir/src.c.obj.d -o CMakeFiles/cmTC_f3b74.dir/src.c.obj -c /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/CMakeFiles/CMakeScratch/TryCompile-gy54Ab/src.c
clang: warning: -Wl,-z,text: 'linker' input unused [-Wunused-command-line-argument]
[2/2] : && /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -L/home/gulfem/fuchsia-idk/arch/x64/lib -fuse-ld=lld CMakeFiles/cmTC_f3b74.dir/src.c.obj -o cmTC_f3b74  -lc && :

This is the source file for these tests:

int main(void) { return 0; }

This bug is specific to PGO builds on AArch64; it does not occur on x86 architectures or in non-PGO builds.

The hang started happening after we adopted IR PGO instead of the Front-End PGO via #156060.

[llvmbot]

@llvm/issue-subscribers-backend-aarch64

Author: None (gulfemsavrun)

We ran into this issue during a multi-stage build where we built our toolchain with PGO. The hang happens specifically after we build Clang with PGO enabled, and then use that PGO-optimized Clang to build the compiler-rt. The linker stalls when compiling some runtime tests like: https://github.com/llvm/llvm-project/blob/597f93d36b035faeb63f4ba0d61a8b8e25eddaab/compiler-rt/cmake/config-ix.cmake#L98

I attached gdb to the stalled lld process and found the following call stack:

#<!-- -->0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S:50
#<!-- -->1  0x0000ffffa4a22658 in __internal_syscall_cancel (a1=a1@<!-- -->entry=281474580353104, a2=a2@<!-- -->entry=393, a3=a3@<!-- -->entry=0, a4=a4@<!-- -->entry=0, a5=a5@<!-- -->entry=0, a6=a6@<!-- -->entry=4294967295, nr=nr@<!-- -->entry=98) at ./nptl/cancellation.c:49
#<!-- -->2  0x0000ffffa4a229f0 in __futex_abstimed_wait_common64 (private=0, futex_word=0xffffe8601050, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#<!-- -->3  __futex_abstimed_wait_common (futex_word=0xffffe8601050, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#<!-- -->4  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@<!-- -->entry=0xffffe8601050, expected=expected@<!-- -->entry=0, clockid=clockid@<!-- -->entry=0, abstime=abstime@<!-- -->entry=0x0, private=private@<!-- -->entry=0)
    at ./nptl/futex-internal.c:139
#<!-- -->5  0x0000ffffa4a253ec in __pthread_cond_wait_common (cond=0xffffe8601030, mutex=0xffffe8601000, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:426
#<!-- -->6  ___pthread_cond_wait (cond=0xffffe8601030, mutex=0xffffe8601000) at ./nptl/pthread_cond_wait.c:458
#<!-- -->7  0x0000aaaaed0f8ea4 in std::__2::__libcpp_condvar_wait[abi:ne220000](pthread_cond_t*, pthread_mutex_t*) (__cv=0xffffe8601050, __m=0x189)
    at /home/gulfem/llvm-project-gulfem/libcxx/include/__thread/support/pthread.h:122
#<!-- -->8  std::__2::condition_variable::wait (this=0xffffe8601050, lk=...) at ../../../llvm-project-gulfem/libcxx/src/condition_variable.cpp:36
#<!-- -->9  0x0000aaaaecfcfa3c in std::__2::condition_variable::wait[abi:nn220000]&lt;llvm::parallel::detail::Latch::sync() const::{lambda()#<!-- -->1}&gt;(std::__2::unique_lock&lt;std::__2::mutex&gt;&amp;, llvm::parallel::detail::Latch::sync() const::{lambda()#<!-- -->1}) (this=0xffffe8601030, __lk=..., __pred=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__condition_variable/condition_variable.h:112
#<!-- -->10 llvm::parallel::detail::Latch::sync (this=0xffffe8600ff8) at llvm/include/llvm/Support/Parallel.h:85
#<!-- -->11 llvm::parallel::TaskGroup::~TaskGroup (this=0xffffe8600ff8) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:191
#<!-- -->12 0x0000aaaae45d6b50 in (anonymous namespace)::Writer&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt;::writeSections (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:3026
#<!-- -->13 (anonymous namespace)::Writer&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt;::run (this=0xffffe8600ea8) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:376
#<!-- -->14 lld::elf::writeResult&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt; (ctx=...) at ../../../../llvm-project-gulfem/lld/ELF/Writer.cpp:100
#<!-- -->15 0x0000aaaae42f1c6c in lld::elf::LinkerDriver::link&lt;llvm::object::ELFType&lt;(llvm::endianness)1, true&gt; &gt; (this=&lt;optimized out&gt;, this@<!-- -->entry=0xaaab2a1035c0, args=...)
    at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:3503
#<!-- -->16 0x0000aaaae42ce1f4 in lld::elf::LinkerDriver::linkerMain (this=&lt;optimized out&gt;, this@<!-- -->entry=0xaaab2a1035c0, argsArr=...) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:729
#<!-- -->17 0x0000aaaae42caba4 in lld::elf::link (args=..., stdoutOS=..., stderrOS=..., exitEarly=&lt;optimized out&gt;, disableOutput=&lt;optimized out&gt;) at ../../../../llvm-project-gulfem/lld/ELF/Driver.cpp:140
#<!-- -->18 0x0000aaaae418d010 in lld::unsafeLldMain (args=..., stdoutOS=..., stderrOS=..., drivers=..., exitEarly=true) at ../../../../llvm-project-gulfem/lld/Common/DriverDispatcher.cpp:163
#<!-- -->19 0x0000aaaae3bcd3d0 in lld_main (argc=argc@<!-- -->entry=32, argv=argv@<!-- -->entry=0xffffe8602f38) at ../../../../llvm-project-gulfem/lld/tools/lld/lld.cpp:90
#<!-- -->20 0x0000aaaae418af4c in findTool (Argc=32, Argv=0xffffe8602f38, Argv0=0xffffe8603e8c "/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/ld.lld")
    at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:68
#<!-- -->21 0x0000aaaae4185830 in main (Argc=32, Argv=0xffffe8602f38) at ../../../../llvm-project-gulfem/llvm/tools/llvm-driver/llvm-driver.cpp:85

I can sometimes reproduce the hang when running tests like the ones mentioned, but the issue is not consistent:

[1/2] /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot -DCOMPILER_RT_HAS_Z_TEXT  --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -MD -MT CMakeFiles/cmTC_f3b74.dir/src.c.obj -MF CMakeFiles/cmTC_f3b74.dir/src.c.obj.d -o CMakeFiles/cmTC_f3b74.dir/src.c.obj -c /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/CMakeFiles/CMakeScratch/TryCompile-gy54Ab/src.c
clang: warning: -Wl,-z,text: 'linker' input unused [-Wunused-command-line-argument]
[2/2] : && /home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/./bin/clang --target=x86_64-unknown-fuchsia --sysroot=/home/gulfem/fuchsia-idk/arch/x64/sysroot --target=x86_64-unknown-fuchsia -I/home/gulfem/fuchsia-idk/pkg/sync/include -I/home/gulfem/fuchsia-idk/pkg/fdio/include -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/home/gulfem/clang-prod-linux-build-relwithdebuginfo/tools/clang/stage2-instrumented-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins=../../../../../../llvm-project-gulfem -ffile-prefix-map=/home/gulfem/llvm-project-gulfem/= -no-canonical-prefixes  --start-no-unused-arguments --unwindlib=none --end-no-unused-arguments -nostdlib++ -nostdinc++ -nodefaultlibs -Wl,-z,text -L/home/gulfem/fuchsia-idk/arch/x64/lib -fuse-ld=lld CMakeFiles/cmTC_f3b74.dir/src.c.obj -o cmTC_f3b74  -lc && :

This is the source file for these tests:

int main(void) { return 0; }

This bug is specific to PGO builds on AArch64; it does not occur on x86 architectures or in non-PGO builds.

The hang started happening after we adopted IR PGO instead of the Front-End PGO via #156060.

[gulfemsavrun]

Here's the stack trace from one of the worker threads:

Thread 17 (Thread 0xffff8910fce0 (LWP 494918) "ld.lld"):
#0  instrumentTargetValueImpl (TargetValue=33, Data=<optimized out>, CounterIndex=<optimized out>, CountValue=1) at ../../../llvm-project-gulfem/compiler-rt/lib/profile/InstrProfilingValue.c:169
#1  __llvm_profile_instrument_target (TargetValue=33, Data=<optimized out>, CounterIndex=<optimized out>) at ../../../llvm-project-gulfem/compiler-rt/lib/profile/InstrProfilingValue.c:244
#2  0x0000aaaace5a2028 in lld::elf::InputSection::writeTo<llvm::object::ELFType<(llvm::endianness)1, true> > (this=0xffff74001dc8, ctx=..., buf=0xffff896250fb "") at ../../../../llvm-project-gulfem/lld/ELF/InputSection.cpp:1326
#3  0x0000aaaace642d5c in lld::elf::OutputSection::writeTo<llvm::object::ELFType<(llvm::endianness)1, true> >(lld::elf::Ctx&, unsigned char*, llvm::parallel::TaskGroup&)::{lambda(unsigned long, unsigned long)#1}::operator()(unsigned long, unsigned long) const (this=0xaaaae048ca00, begin=<optimized out>, end=1) at ../../../../llvm-project-gulfem/lld/ELF/OutputSections.cpp:554
#4  0x0000aaaad71a1b4c in std::__2::__function::__policy_func<void ()>::operator()[abi:nn220000]() const (this=0xaaaae048ca50) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__functional/function.h:502
#5  std::__2::function<void()>::operator() (this=0xaaaae048ca50) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__functional/function.h:754
#6  llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0::operator()() const (this=0xaaaae048ca50) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:199
#7  std::__2::__invoke[abi:nn220000]<llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&>(llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&) (__args=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__type_traits/invoke.h:90
#8  std::__2::__invoke_void_return_wrapper<void, true>::__call[abi:nn220000]<llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&>(llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&) (__args=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__type_traits/invoke.h:350
#9  std::__2::__invoke_r[abi:nn220000]<void, llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&>(llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0&) (__args=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__type_traits/invoke.h:356
#10 std::__2::__function::__policy_func<void ()>::__call_func[abi:nn220000]<llvm::parallel::TaskGroup::spawn(std::__2::function<void ()>)::$_0>(std::__2::__function::__policy_storage const*) (__buf=__buf@entry=0xffff8910f320) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__functional/function.h:443
#11 0x0000aaaad71a13a8 in std::__2::__function::__policy_func<void ()>::operator()[abi:nn220000]() const (this=0xffff8910f320) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__functional/function.h:502
#12 std::__2::function<void()>::operator() (this=0xffff8910f320) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__functional/function.h:754
#13 llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::work (this=this@entry=0xaaaad9c432c0 <llvm::parallel::detail::(anonymous namespace)::Executor::getDefaultExecutor()::Exec>, S=..., ThreadID=ThreadID@entry=2299589568) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:122
#14 0x0000aaaad71a114c in llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}::operator()() const (this=<optimized out>) at ../../../../llvm-project-gulfem/llvm/lib/Support/Parallel.cpp:68
#15 std::__2::__invoke[abi:nn220000]<llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}>(llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}&&) (__args=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__type_traits/invoke.h:90
#16 std::__2::__thread_execute[abi:nn220000]<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}, , 0ul>(std::__2::tuple<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}>&, std::__2::__integer_sequence<unsigned long, 0ul>) (__t=...) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__thread/thread.h:159
#17 std::__2::__thread_proxy[abi:nn220000]<std::__2::tuple<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::{lambda()#1}> >(void*) (__vp=<optimized out>) at /home/gulfem/clang-prod-linux-build-relwithdebuginfo/./bin/../include/c++/v1/__thread/thread.h:167
#18 0x0000ffff89265f78 in start_thread (arg=0xffff8910fce0) at ./nptl/pthread_create.c:448
#19 0x0000ffff892cde8c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone3.S:76

The concurrency bug that triggers the thread deadlock is located in the value profiling code of the compiler-rt runtime:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/profile/InstrProfilingValue.c#L169

When lld runs in parallel, it causes the linked code (including compiler-rt's instrumentation) to execute its own logic on multiple threads. Since the value profiling code is not thread-safe, this concurrent access results in the hang. The issue is fundamentally a compiler-rt problem, not an lld bug.

@david-xl as one of the original author's of value profiling, could you please confirm if concurrency was considered in the design of the value profiling structures (like the ValueProfNode linked list)? Our analysis suggests thread safety was not accounted for.

Short-Term Unblock Strategy

1. Disable Value Profiling: Temporarily disable the value profiling to avoid the concurrency issue when we build PGO-instrumented Clang
2. Disable Parallelism: Continue building with the PGO-instrumented Clang, but force lld to use a single thread (e.g., using the --threads=1 flag). This prevents the simultaneous access that causes the deadlock.

Long-Term Solution: Implement Thread Safety
Ensure the profiling data structures are thread-safe and can be safely accessed by multiple threads simultaneously.