libarcher pre-merge checks failing

[Meinersbur]

The current Phabricator pre-merge tests are failing because of libarcher tests failing. Phabricator does not show the actual failing tests, just that the Debian build failed. Opening the log file reveals which tests are failing

https://buildkite.com/llvm-project/premerge-checks/builds/76168#81ae5887-ac45-49eb-a03b-ce6be1f38657 of https://reviews.llvm.org/D118357

FAIL: libarcher :: races/critical-unrelated.c (91 of 317)
******************** TEST 'libarcher :: races/critical-unrelated.c' FAILED ********************
Script:
--
: 'RUN: at line 13';   /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -fopenmp -pthread -fno-experimental-isel -g -O1 -fsanitize=thread  -I /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests -I /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/runtime/src -L /var/lib/buildkite-agent/builds/llvm-project/build/lib -Wl,-rpath,/var/lib/buildkite-agent/builds/llvm-project/build/lib   /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic && env TSAN_OPTIONS='ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1' /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/deflake.bash /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp 2>&1 | tee /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log | /var/lib/buildkite-agent/builds/llvm-project/build/./bin/FileCheck /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
: 'RUN: at line 14';   /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -fopenmp -pthread -fno-experimental-isel -g -O1 -fsanitize=thread  -I /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests -I /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/runtime/src -L /var/lib/buildkite-agent/builds/llvm-project/build/lib -Wl,-rpath,/var/lib/buildkite-agent/builds/llvm-project/build/lib   /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp -latomic && env ARCHER_OPTIONS="ignore_serial=1 report_data_leak=1" env TSAN_OPTIONS='ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1' /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/deflake.bash /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp 2>&1 | tee /var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log | /var/lib/buildkite-agent/builds/llvm-project/build/./bin/FileCheck /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 13"
$ "/var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang" "-fopenmp" "-pthread" "-fno-experimental-isel" "-g" "-O1" "-fsanitize=thread" "-I" "/var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests" "-I" "/var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/runtime/src" "-L" "/var/lib/buildkite-agent/builds/llvm-project/build/lib" "-Wl,-rpath,/var/lib/buildkite-agent/builds/llvm-project/build/lib" "/var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c" "-o" "/var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp" "-latomic"
$ "env" "TSAN_OPTIONS=ignore_noninstrumented_modules=0:ignore_noninstrumented_modules=1" "/var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/deflake.bash" "/var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp"
$ "tee" "/var/lib/buildkite-agent/builds/llvm-project/build/projects/openmp/tools/archer/tests/races/Output/critical-unrelated.c.tmp.log"
$ "/var/lib/buildkite-agent/builds/llvm-project/build/./bin/FileCheck" "/var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c"
# command stderr:
/var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c:37:16: error: CHECK-NEXT: expected string not found in input
// CHECK-NEXT: #0 {{.*}}critical-unrelated.c:29
               ^
<stdin>:4:17: note: scanning from here
 Write of size 4 at 0x7ffe13e37c94 by thread T3:
                ^
<stdin>:22:29: note: possible intended match here
SUMMARY: ThreadSanitizer: data race critical-unrelated.c:? in .omp_outlined.
                            ^

Input file: <stdin>
Check file: /var/lib/buildkite-agent/builds/llvm-project/openmp/tools/archer/tests/races/critical-unrelated.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           1: /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25 
           2: ================== 
           3: WARNING: ThreadSanitizer: data race (pid=2868172) 
           4:  Write of size 4 at 0x7ffe13e37c94 by thread T3: 
next:37'0                     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
           5:  #0 .omp_outlined. critical-unrelated.c:? (critical-unrelated.c.tmp+0x4ceaf1) 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           6:  #1 __kmp_invoke_microtask ??:? (libomp.so+0xde772) (BuildId: 9b2d0c31f22855b5) 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           7:  #2 main ??:? (critical-unrelated.c.tmp+0x4cea68) 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           8:  
next:37'0     ~
           9:  Previous write of size 4 at 0x7ffe13e37c94 by main thread: 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .
          17:  
next:37'0     ~
          18:  Thread T3 (tid=2868646, running) created by main thread at: 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          19:  #0 pthread_create /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1017 (critical-unrelated.c.tmp+0x45070d) 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          20:  #1 __kmp_create_worker /var/lib/buildkite-agent/builds/llvm-project/openmp/runtime/src/z_Linux_util.cpp:803 (libomp.so+0xbe036) (BuildId: 9b2d0c31f22855b5) 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          21:  
next:37'0     ~
          22: SUMMARY: ThreadSanitizer: data race critical-unrelated.c:? in .omp_outlined. 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
next:37'1                                 ?                                                 possible intended match
          23: ================== 
next:37'0     ~~~~~~~~~~~~~~~~~~~
          24: DONE 
next:37'0     ~~~~~
          25: ThreadSanitizer: reported 1 warnings 
next:37'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>

error: command failed with exit status: 1

********************

Failed Tests (9):
  libarcher :: races/critical-unrelated.c
  libarcher :: races/lock-nested-unrelated.c
  libarcher :: races/lock-unrelated.c
  libarcher :: races/parallel-simple.c
  libarcher :: races/task-dependency.c
  libarcher :: races/task-taskgroup-unrelated.c
  libarcher :: races/task-taskwait-nested.c
  libarcher :: races/task-two.c
  libarcher :: task/task_late_fulfill.c

Or without any patch applied:
https://buildkite.com/llvm-project/llvm-main/builds/4176#78b9c64c-b3f0-494a-ac5d-6b45334e8e30

[llvmbot]

@llvm/issue-subscribers-openmp

[jprotze]

5: #0 .omp_outlined. critical-unrelated.c:? (critical-unrelated.c.tmp+0x4ceaf1)

Any suggestion how to deal with debian failing to report the line number? I wonder, why the compiler-rt/tsan tests don't have the same issue. The tsan tests also match the reports for line numbers:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/tsan/simple_stack.c#L61

I cannot see a significant difference in the compiler flags configured for the tsan tests.

[Meinersbur]

Does this line have to do with it?

/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25 

Maybe PATH is different such that the sanitizer tests use llvm-symbolizer instead?

[Meinersbur]

CC @metaflow (for buildkite bots) @vitalybuka (who recently worked on compiler-rt symbolizer support)

[pogo59]

/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25

This happens because the default DWARF version for Clang was recently changed to DWARF v5, and the addr2line tool isn't adequately prepared.
Check with @dwblaikie

[dwblaikie]

Yep, that all sounds right to me - symbolizer uses llvm-symbolizer which has DWARFv5 (well, support for the ways LLVM generates DWARFv5 which GCC doesn't (strx form encodings in particular)) which the version of addr2line you're using does not?

[llvmbot]

@llvm/issue-subscribers-infrastructure

[AaronBallman]

As this is currently causing all CI pipelines to fail, I think we should disable the libarcher tests in precommit CI again.

google/llvm-premerge-checks#323 (comment)

[asl]

Tagging @ChristianKuehnel @metaflow

[jprotze]

As this is currently causing all CI pipelines to fail, I think we should disable the libarcher tests in precommit CI again.

google/llvm-premerge-checks#323 (comment)

Similarly, I could argue: take the debian pipelines off, as long as the machines are not updated to support DWARF5 in addr2line.

Instead, I suggest to enforce dwarf-4 symbols for the libarcher tests.

[pogo59]

Instead, I suggest to enforce dwarf-4 symbols for the libarcher tests.

This seems like a better way forward to me.