[lit] Expand late substitutions before running builtins

[boomanaiden154]

This enables the use of readfile substitutions for populating
environment variables. This is necessary in some compiler-rt tests.

[llvmbot]

@llvm/pr-subscribers-testing-tools

Author: Aiden Grossman (boomanaiden154)

Changes

This enables the use of readfile substitutions for populating
environment variables. This is necessary in some compiler-rt tests.

---

Full diff: https://github.com/llvm/llvm-project/pull/165140.diff

4 Files Affected:

• (modified) llvm/utils/lit/lit/TestRunner.py (+4-3)
• (added) llvm/utils/lit/tests/Inputs/shtest-readfile/env.txt (+6)
• (modified) llvm/utils/lit/tests/shtest-readfile-external.py (+1-1)
• (modified) llvm/utils/lit/tests/shtest-readfile.py (+5-1)

diff --git a/llvm/utils/lit/lit/TestRunner.py b/llvm/utils/lit/lit/TestRunner.py
index be01f15d649a1..999616f891b9c 100644
--- a/llvm/utils/lit/lit/TestRunner.py
+++ b/llvm/utils/lit/lit/TestRunner.py
@@ -825,6 +825,10 @@ def _executeShCmd(cmd, shenv, results, timeoutHelper):
         not_args = []
         not_count = 0
         not_crash = False
+
+        # Expand all late substitutions.
+        args = _expandLateSubstitutions(j, args, cmd_shenv.cwd)
+
         while True:
             if args[0] == "env":
                 # Create a copy of the global environment and modify it for
@@ -874,9 +878,6 @@ def _executeShCmd(cmd, shenv, results, timeoutHelper):
         # Ensure args[0] is hashable.
         args[0] = expand_glob(args[0], cmd_shenv.cwd)[0]
 
-        # Expand all late substitutions.
-        args = _expandLateSubstitutions(j, args, cmd_shenv.cwd)
-
         inproc_builtin = inproc_builtins.get(args[0], None)
         if inproc_builtin and (args[0] != "echo" or len(cmd.commands) == 1):
             # env calling an in-process builtin is useless, so we take the safe
diff --git a/llvm/utils/lit/tests/Inputs/shtest-readfile/env.txt b/llvm/utils/lit/tests/Inputs/shtest-readfile/env.txt
new file mode 100644
index 0000000000000..c790b687c4364
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/shtest-readfile/env.txt
@@ -0,0 +1,6 @@
+## Tests that readfile works with the env builtin.
+# RUN: echo -n "hello" > %t.1
+# RUN: env TEST=%{readfile:%t.1} python3 -c "import os; print(os.environ['TEST'])"
+
+## Fail the test so we can assert on the output.
+# RUN: not echo return
\ No newline at end of file
diff --git a/llvm/utils/lit/tests/shtest-readfile-external.py b/llvm/utils/lit/tests/shtest-readfile-external.py
index c00bff45c8703..6fe1088efd674 100644
--- a/llvm/utils/lit/tests/shtest-readfile-external.py
+++ b/llvm/utils/lit/tests/shtest-readfile-external.py
@@ -6,7 +6,7 @@
 # UNSUPPORTED: system-windows
 # RUN: env LIT_USE_INTERNAL_SHELL=0 not %{lit} -a -v %{inputs}/shtest-readfile | FileCheck -match-full-lines -DTEMP_PATH=%S/Inputs/shtest-readfile/Output %s
 
-# CHECK: -- Testing: 4 tests{{.*}}
+# CHECK: -- Testing: 5 tests{{.*}}
 
 # CHECK-LABEL: FAIL: shtest-readfile :: absolute-paths.txt ({{[^)]*}})
 # CHECK: echo $(cat [[TEMP_PATH]]/absolute-paths.txt.tmp) && test -e [[TEMP_PATH]]/absolute-paths.txt.tmp {{.*}}
diff --git a/llvm/utils/lit/tests/shtest-readfile.py b/llvm/utils/lit/tests/shtest-readfile.py
index 66e3a042bf787..be3915a1608b4 100644
--- a/llvm/utils/lit/tests/shtest-readfile.py
+++ b/llvm/utils/lit/tests/shtest-readfile.py
@@ -5,12 +5,16 @@
 
 # RUN: env LIT_USE_INTERNAL_SHELL=1  not %{lit} -a -v %{inputs}/shtest-readfile | FileCheck -match-full-lines -DTEMP_PATH=%S%{fs-sep}Inputs%{fs-sep}shtest-readfile%{fs-sep}Output %s
 
-# CHECK: -- Testing: 4 tests{{.*}}
+# CHECK: -- Testing: 5 tests{{.*}}
 
 # CHECK-LABEL: FAIL: shtest-readfile :: absolute-paths.txt ({{[^)]*}})
 # CHECK: echo hello
 # CHECK: # executed command: echo '%{readfile:[[TEMP_PATH]]{{[\\\/]}}absolute-paths.txt.tmp}'
 
+# CHECK-LABEL: FAIL: shtest-readfile :: env.txt ({{[^)]*}})
+# CHECK: env TEST=hello python3 -c "import os; print(os.environ['TEST'])"
+# CHECK: # | hello
+
 # CHECK-LABEL: FAIL: shtest-readfile :: file-does-not-exist.txt ({{[^)]*}})
 # CHECK: # executed command: @echo 'echo %{readfile:/file/does/not/exist}'
 # CHECK: # | File specified in readfile substitution does not exist: {{.*}}/file/does/not/exist

[pawosm-arm]

I've tried this locally, and it seems it does really help with the issue I'm facing while building our toolchain, so I'm interested in landing this. One thing: the way this PR has been opened does not make it easy to just cherry-pick it, I had to rework this patch manually in order to try it out.

[pawosm-arm]

fixes the issue I'm facing