Reapply "[compiler-rt] Default to Lit's Internal Shell"

[boomanaiden154]

This reverts commit 8cc49fb.

This was causing failures on two specific buildbots that have since been fixed.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Aiden Grossman (boomanaiden154)

Changes

This reverts commit 8cc49fb.

This was causing failures on two specific buildbots that have since been fixed.

---

Full diff: https://github.com/llvm/llvm-project/pull/168232.diff

2 Files Affected:

• (modified) compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cpp (+4-2)
• (modified) compiler-rt/test/lit.common.cfg.py (+5-3)

diff --git a/compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cpp b/compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cpp
index f60a6a4ef79e6..1daf56d9d0a52 100644
--- a/compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cpp
+++ b/compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cpp
@@ -23,9 +23,11 @@
 // RUN:   | FileCheck %s --check-prefixes=CHECK-REALLOC,CHECK-CRASH
 // RUN: %env_asan_opts=allocator_may_return_null=1     %run %t realloc 2>&1 \
 // RUN:   | FileCheck %s --check-prefixes=CHECK-REALLOC,CHECK-NULL
-// RUN: %env_asan_opts=allocator_may_return_null=0 not %run %t realloc-after-malloc 2>&1 \
+// RUN: %export_asan_opts=allocator_may_return_null=0
+// RUN: not %run %t realloc-after-malloc 2>&1 \
 // RUN:   | FileCheck %s --check-prefixes=CHECK-MALLOC-REALLOC,CHECK-CRASH
-// RUN: %env_asan_opts=allocator_may_return_null=1     %run %t realloc-after-malloc 2>&1 \
+// RUN: %export_asan_opts=allocator_may_return_null=1
+// RUN: %run %t realloc-after-malloc 2>&1 \
 // RUN:   | FileCheck %s --check-prefixes=CHECK-MALLOC-REALLOC,CHECK-NULL
 
 // ASan shadow memory on s390 is too large for this test.
diff --git a/compiler-rt/test/lit.common.cfg.py b/compiler-rt/test/lit.common.cfg.py
index 3f7dd8e402b78..0f0f87915bafe 100644
--- a/compiler-rt/test/lit.common.cfg.py
+++ b/compiler-rt/test/lit.common.cfg.py
@@ -113,6 +113,9 @@ def push_dynamic_library_lookup_path(config, new_path):
         config.environment[dynamic_library_lookup_var] = new_ld_library_path_64
 
 
+# TODO: Consolidate the logic for turning on the internal shell by default for all LLVM test suites.
+# See https://github.com/llvm/llvm-project/issues/106636 for more details.
+#
 # Choose between lit's internal shell pipeline runner and a real shell.  If
 # LIT_USE_INTERNAL_SHELL is in the environment, we use that as an override.
 use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
@@ -120,9 +123,8 @@ def push_dynamic_library_lookup_path(config, new_path):
     # 0 is external, "" is default, and everything else is internal.
     execute_external = use_lit_shell == "0"
 else:
-    # Otherwise we default to internal on Windows and external elsewhere, as
-    # bash on Windows is usually very slow.
-    execute_external = not sys.platform in ["win32"]
+    # Otherwise we default to internal everywhere.
+    execute_external = False
 
 # Allow expanding substitutions that are based on other substitutions
 config.recursiveExpansionLimit = 10

[boomanaiden154]

It looks like the ppc64le failures were never real in the first place. The bot seems to fail ~50% of its runs on some compiler-rt test or another.

[DanBlackwell]

I think this commit has caused a number of test failures on Green Dragon (Apple OS CI testing): https://green.lab.llvm.org/job/llvm.org/job/clang-stage1-RA/5985/.

[dyung]

Our internal linux builder is also hitting issues in a bunch of XRay tests that start passing when I set LIT_USE_INTERNAL_SHELL=0. Errors look similar to this:

FAIL: XRay-x86_64-linux :: TestCases/Posix/c-test.cpp (3 of 37)                                                                                                                                                                        
******************** TEST 'XRay-x86_64-linux :: TestCases/Posix/c-test.cpp' FAILED ********************                                                                                                                                
Exit Code: 127                                                                                                                                                                                                                         
                                                                                                                                                                                                                                       
Command Output (stdout):                                                                                                                                                                                                               
--                                                                                                                                                                                                                                     
# RUN: at line 1                                                                                                                                                                                                                       
/home/dyung/src/git/merge/build-test/./bin/clang  -fxray-instrument  -m64  -g -fxray-modes=xray-basic,xray-fdr,xray-profiling -o /home/dyung/src/git/merge/build-test/projects/compiler-rt/test/xray/X86_64LinuxConfig/TestCases/Posix/
Output/c-test.cpp.tmp /home/dyung/src/git/merge/compiler-rt/test/xray/TestCases/Posix/c-test.cpp                                                                                                                                       
# executed command: /home/dyung/src/git/merge/build-test/./bin/clang -fxray-instrument -m64 -g -fxray-modes=xray-basic,xray-fdr,xray-profiling -o /home/dyung/src/git/merge/build-test/projects/compiler-rt/test/xray/X86_64LinuxConfig
/TestCases/Posix/Output/c-test.cpp.tmp /home/dyung/src/git/merge/compiler-rt/test/xray/TestCases/Posix/c-test.cpp                                                                                                                      
# RUN: at line 2                                                                                                                                                                                                                       
rm -f xray-log.c-test.*                                                                                                                                                                                                                
# executed command: rm -f 'xray-log.c-test.*'                                                                                                                                                                                          
# RUN: at line 3                                                                                                                                                                                                                       
XRAY_OPTIONS=patch_premain=true:verbosity=1:xray_mode=xray-basic /home/dyung/src/git/merge/build-test/projects/compiler-rt/test/xray/X86_64LinuxConfig/TestCases/Posix/Output/c-test.cpp.tmp      2>&1 | FileCheck /home/dyung/src/git/
merge/compiler-rt/test/xray/TestCases/Posix/c-test.cpp                                                                                                                                                                                 
# executed command: XRAY_OPTIONS=patch_premain=true:verbosity=1:xray_mode=xray-basic /home/dyung/src/git/merge/build-test/projects/compiler-rt/test/xray/X86_64LinuxConfig/TestCases/Posix/Output/c-test.cpp.tmp                       
# .---command stderr------------                                                                                                                                                                                                       
# | 'XRAY_OPTIONS=patch_premain=true:verbosity=1:xray_mode=xray-basic': command not found                                                                                                                                              
# `-----------------------------                                                                                                                                                                                                       
# error: command failed with exit status: 127

[DanBlackwell]

@boomanaiden154 I can see some of the Darwin tests for sanitizers using !(which sometool) and backticks. Looking at the RFC, the lit shell will not aim to support these. What would be considered an appropriate workaround? sh -c?

[boomanaiden154]

the lit shell will not aim to support these. What would be considered an appropriate workaround? sh -c?

No. They need to be rewritten to not use subshells. I'm working on a fix and hope to have patches submitted soon. Working on the XRay failures first that I expect to have done in ~30 minutes.

[boomanaiden154]

XRay tests should be fixed as of c7a9be8. #168378 actually gets them running inside the bootstrap build since they were incorrectly configured before.

[boomanaiden154]

Reverted for now in cf0909e. I'll hopefully have patches up to get the tests fixed later today.

[DanBlackwell]

Hi @boomanaiden154, it looks like a good number of these tests failing are specific to Darwin - if you need any help or have a patch to review for these could you please tag me, @ndrewh, @wrotki or @thetruestblue? Thanks

[boomanaiden154]

Minus the XRay tests, all the failures reported here are on Darwin. Hoping to put patches up later today to get things working (will ping you all for review) and will then reland the enablement once those land.

[zeroomega]

We also see test issues after this patch landed:

UNRESOLVED: Profile-x86_64 :: Linux/instrprof-debug-info-correlate-debuginfod.c (2512 of 18117)
******************** TEST 'Profile-x86_64 :: Linux/instrprof-debug-info-correlate-debuginfod.c' FAILED ********************
Exception during script execution:
Traceback (most recent call last):
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/worker.py", line 77, in _execute_test_handle_errors
    result = test.config.test_format.execute(test, lit_config)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 29, in execute
    return lit.TestRunner.executeShTest(
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 2497, in executeShTest
    return _runShTest(test, litConfig, useExternalSh, script, tmpBase)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 2419, in _runShTest
    res = runOnce(execdir)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 2395, in runOnce
    res = executeScriptInternal(
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1229, in executeScriptInternal
    exitCode, timeoutInfo = executeShCmd(
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 212, in executeShCmd
    finalExitCode = _executeShCmd(cmd, shenv, results, timeoutHelper)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 784, in _executeShCmd
    res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 784, in _executeShCmd
    res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 784, in _executeShCmd
    res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
  [Previous line repeated 1 more time]
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 789, in _executeShCmd
    res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 789, in _executeShCmd
    res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 930, in _executeShCmd
    stdin, stdout, stderr = processRedirects(
  File "/b/s/w/ir/x/w/llvm-llvm-project/llvm/utils/lit/lit/TestRunner.py", line 724, in processRedirects
    fd = open(redir_filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: b'/b/s/w/ir/x/w/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/profile/Profile-x86_64/Linux/(llvm-profdata'


********************

Failed build task: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8698056561481521121/overview

Could we revert this PR?

[boomanaiden154]

It was reverted four hours ago. cf0909e

[mysterymath]

It was reverted four hours ago. cf0909e

Hm? The commit that lands this PR is still the most recent one to touch the affected file: https://github.com/llvm/llvm-project/commits/main/compiler-rt/test/lit.common.cfg.py

[boomanaiden154]

Ah, yeah. Looks like I only pushed that one to a branch instead of main. 🙃

Actually pushed now as eb20b53.

[mysterymath]

Thanks!