[OpenMP] Some libarcher tests fail on Ubuntu 22.04 when llvm-symbolizer is not found

[luporl]

Failed tests:

libarcher :: races/critical-unrelated.c
libarcher :: races/lock-nested-unrelated.c
libarcher :: races/lock-unrelated.c
libarcher :: races/parallel-simple.c
libarcher :: races/task-dependency.c
libarcher :: races/task-taskgroup-unrelated.c
libarcher :: races/task-taskwait-nested.c
libarcher :: races/task-two.c
libarcher :: races/taskwait-depend.c
libarcher :: task/task_late_fulfill.c

The failures in the tests above are the same. They use tsan and expect the addresses to have been converted to lines in its report.
But the CI machine doesn't have llvm-symbolizer available in PATH, so addr2line is used instead, but the Ubuntu 22.04 version doesn't support the DWARF used in the test programs:
addr2line: DWARF error: invalid or unhandled FORM value: 0x23

******************** TEST 'libarcher :: races/critical-unrelated.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 13
/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/clang -fopenmp  -gdwarf-4 -O1 -fsanitize=thread  -I /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests -I /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -L /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src   /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic && env TSAN_OPTIONS='ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1' /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/deflake.bash /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp 2>&1 | tee /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log | /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/FileCheck /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# executed command: /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/clang -fopenmp -gdwarf-4 -O1 -fsanitize=thread -I /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests -I /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -L /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic
# executed command: env TSAN_OPTIONS=ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1 /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/deflake.bash /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp
# executed command: tee /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log
# executed command: /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/FileCheck /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# .---command stderr------------
# | /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c:37:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: #0 {{.*}}critical-unrelated.c:29
# |                ^
# | <stdin>:4:17: note: scanning from here
# |  Write of size 4 at 0x7fff51c01cac by thread T4:
# |                 ^
# | <stdin>:5:17: note: possible intended match here
# |  #0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405)
# |                 ^
# | 
# | Input file: <stdin>
# | Check file: /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23 
# |            2: ================== 
# |            3: WARNING: ThreadSanitizer: data race (pid=1018671) 
# |            4:  Write of size 4 at 0x7fff51c01cac by thread T4: 
# | next:37'0                     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            5:  #0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:37'1                     ?                                                                possible intended match
# |            6:  #1 __kmp_invoke_microtask ??:? (libomp.so+0xec5d8) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            7:  #2 main ??:? (critical-unrelated.c.tmp+0x30393) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            8:  
# | next:37'0     ~
# |            9:  Previous write of size 4 at 0x7fff51c01cac by main thread: 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           10:  #0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            .
# |            .
# |            .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

The failure doesn't happen on Ubuntu 24.04, that has a more recent version of addr2line.
But as Ubuntu 22.04 is still used for releases, upgrading to 24.04 doesn't seem an option.

When llvm-symbolizer is available in PATH the issue also doesn't occur. Requiring another LLVM version to be installed and available in PATH doesn't seem reasonable, but maybe libarcher tests could adjust PATH instead, to use the just built version of llvm-symbolizer.

Another option could be changing the tests to use an older DWARF version, compatible with older addr2line.

More details at #160581 and https://github.com/llvm/llvm-project/actions/runs/19738643117/job/56556800778?pr=160581.

[llvmbot]

@llvm/issue-subscribers-openmp

Author: Leandro Lupori (luporl)

Failed tests: ``` libarcher :: races/critical-unrelated.c libarcher :: races/lock-nested-unrelated.c libarcher :: races/lock-unrelated.c libarcher :: races/parallel-simple.c libarcher :: races/task-dependency.c libarcher :: races/task-taskgroup-unrelated.c libarcher :: races/task-taskwait-nested.c libarcher :: races/task-two.c libarcher :: races/taskwait-depend.c libarcher :: task/task_late_fulfill.c ```

The failures in the tests above are the same. They use tsan and expect the addresses to have been converted to lines in its report.
But the CI machine doesn't have llvm-symbolizer available in PATH, so addr2line is used instead, but the Ubuntu 22.04 version doesn't support the DWARF used in the test programs:
addr2line: DWARF error: invalid or unhandled FORM value: 0x23

******************** TEST 'libarcher :: races/critical-unrelated.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 13
/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/clang -fopenmp  -gdwarf-4 -O1 -fsanitize=thread  -I /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests -I /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -L /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src   /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic && env TSAN_OPTIONS='ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1' /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/deflake.bash /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp 2>&1 | tee /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log | /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/FileCheck /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# executed command: /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/clang -fopenmp -gdwarf-4 -O1 -fsanitize=thread -I /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests -I /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -L /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/runtime/src /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic
# executed command: env TSAN_OPTIONS=ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1 /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/deflake.bash /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp
# executed command: tee /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/runtimes/runtimes-bins/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log
# executed command: /mnt/build/tools/clang/stage2-instrumented-bins/tools/clang/stage2-bins/./bin/FileCheck /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# .---command stderr------------
# | /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c:37:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: #<!-- -->0 {{.*}}critical-unrelated.c:29
# |                ^
# | &lt;stdin&gt;:4:17: note: scanning from here
# |  Write of size 4 at 0x7fff51c01cac by thread T4:
# |                 ^
# | &lt;stdin&gt;:5:17: note: possible intended match here
# |  #<!-- -->0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405)
# |                 ^
# | 
# | Input file: &lt;stdin&gt;
# | Check file: /home/runner/work/llvm-project/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | &lt;&lt;&lt;&lt;&lt;&lt;
# |            1: /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23 
# |            2: ================== 
# |            3: WARNING: ThreadSanitizer: data race (pid=1018671) 
# |            4:  Write of size 4 at 0x7fff51c01cac by thread T4: 
# | next:37'0                     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            5:  #<!-- -->0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:37'1                     ?                                                                possible intended match
# |            6:  #<!-- -->1 __kmp_invoke_microtask ??:? (libomp.so+0xec5d8) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            7:  #<!-- -->2 main ??:? (critical-unrelated.c.tmp+0x30393) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            8:  
# | next:37'0     ~
# |            9:  Previous write of size 4 at 0x7fff51c01cac by main thread: 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           10:  #<!-- -->0 main.omp_outlined critical-unrelated.c:? (critical-unrelated.c.tmp+0xe9405) 
# | next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            .
# |            .
# |            .
# | &gt;&gt;&gt;&gt;&gt;&gt;
# `-----------------------------
# error: command failed with exit status: 1

The failure doesn't happen on Ubuntu 24.04, that has a more recent version of addr2line.
But as Ubuntu 22.04 is still used for releases, upgrading to 24.04 doesn't seem an option.

When llvm-symbolizer is available in PATH the issue also doesn't occur. Requiring another LLVM version to be installed and available in PATH doesn't seem reasonable, but maybe libarcher tests could adjust PATH instead, to use the just built version of llvm-symbolizer.

Another option could be changing the tests to use an older DWARF version, compatible with older addr2line.

More details at #160581 and https://github.com/llvm/llvm-project/actions/runs/19738643117/job/56556800778?pr=160581.

[jprotze]

As the output for the RUN in line 13 clearly shows, the tests are already built with -gdwarf-4 for exactly the sketched reason. I don't quite understand the issue here. I successfully tested OpenMP on an Ubuntu 22.04 system before I upgraded to 24.04.

If you build the complete llvm-project anyways, you can just append the install/bin directory to the PATH variable.

[luporl]

As the output for the RUN in line 13 clearly shows, the tests are already built with -gdwarf-4 for exactly the sketched reason. I don't quite understand the issue here. I successfully tested OpenMP on an Ubuntu 22.04 system before I upgraded to 24.04.

I don't know much about DWARF, but my searches for FORM 0x23 showed that it corresponds to DWARF v5 DW_FORM_rnglistx, which means that somehow some DWARF v5 elements are present in the final binary.
I'll try to find where this is coming from.

This explains addr2line error message:
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23

I can easily reproduce this issue on Ubuntu 22.04, by building clang and OpenMP and running
ninja -C runtimes/runtimes-bins check-libarcher

You just can't have llvm-symbolizer in PATH.
This issue wasn't detected by Linaro's buildbots because they have LLVM installed in /usr.

If you build the complete llvm-project anyways, you can just append the install/bin directory to the PATH variable.

The problem is that this issue is happening on Build Release Binaries (ubuntu-22.04) / Build Release Package and Build Release Binaries (ubuntu-22.04-arm) / BuildRelease Package CI jobs. They run the tests without installing LLVM, just like LLVM buildbots. AFAIU these jobs are also used for the official LLVM releases.

I find it strange to require LLVM to be installed and in PATH just for libarcher tests to pass, since all other tests are passing.

[luporl]

It looks like DWARF v5 DW_FORM_rnglistx is coming from libclang_rt.tsan.a, according to llvm-dwarfdump --show-form:

0x00016bda: DW_TAG_compile_unit
              DW_AT_producer [DW_FORM_strx1]    ("clang version 22.0.0git (git@github.com:llvm/llvm-project.git 63ea3537d55f75be0d6fb92aef16465b291fa9ed)")
              DW_AT_language [DW_FORM_data2]    (DW_LANG_C_plus_plus_14)
              DW_AT_name [DW_FORM_strx1]        ("/home/leandro.lupori/git/flang/llvm-project/compiler-rt/lib/tsan/rtl/tsan_debugging.cpp")
              DW_AT_str_offsets_base [DW_FORM_sec_offset]       (0x00002a60)
              DW_AT_stmt_list [DW_FORM_sec_offset]      (0x00024d0c)
              DW_AT_comp_dir [DW_FORM_strx1]    ("/home/leandro.lupori/git/flang/build/runtimes/runtimes-bins")
              DW_AT_low_pc [DW_FORM_addr]       (0x0000000000000000)
              DW_AT_ranges [DW_FORM_rnglistx]   (indexed (0x35) rangelist = 0x000094cc
                 [0x0000000000063a00, 0x0000000000063a04)
                 [0x0000000000063a04, 0x0000000000063a20)
...

The test source doesn't use it:

0x00067a2a: DW_TAG_compile_unit
              DW_AT_producer [DW_FORM_strp]     ("clang version 22.0.0git (git@github.com:llvm/llvm-project.git 63ea3537d55f75be0d6fb92aef16465b291fa9ed)")
              DW_AT_language [DW_FORM_data2]    (DW_LANG_C11)
              DW_AT_name [DW_FORM_strp] ("/home/leandro.lupori/git/flang/llvm-project/openmp/tools/archer/tests/races/lock-unrelated.c")
              DW_AT_stmt_list [DW_FORM_sec_offset]      (0x000a2919)
              DW_AT_comp_dir [DW_FORM_strp]     ("/home/leandro.lupori/git/flang/build/runtimes/runtimes-bins")
              DW_AT_low_pc [DW_FORM_addr]       (0x0000000000000000)
              DW_AT_ranges [DW_FORM_sec_offset] (0x00000000
                 [0x0000000000039940, 0x0000000000039a10)
                 [0x00000000000fa83c, 0x00000000000fa890))

So it seems -gdwarf-4 is being applied only the test source, but not to other statically linked libraries.