[utils] remove flakiness in verbosity.py test case #167801

hnrklssn · 2025-11-13T01:04:48Z

The rate at which lit's progress bar progresses is not deterministic. Don't try to match the contents of the progress bar, only the fact that it's there.

llvmbot · 2025-11-13T01:05:22Z

@llvm/pr-subscribers-testing-tools

Author: Henrik G. Olsson (hnrklssn)

Changes

The rate at which lit's progress bar progresses is not deterministic. Don't try to match the contents of the progress bar, only the fact that it's there.

Full diff: https://github.com/llvm/llvm-project/pull/167801.diff

1 Files Affected:

(modified) llvm/utils/lit/tests/verbosity.py (+26-26)

diff --git a/llvm/utils/lit/tests/verbosity.py b/llvm/utils/lit/tests/verbosity.py
index 9b1690695d392..62baf618e2aca 100644
--- a/llvm/utils/lit/tests/verbosity.py
+++ b/llvm/utils/lit/tests/verbosity.py
@@ -47,9 +47,9 @@
 # SUCCINCT:      -- Testing: 5 tests, 1 workers --
 # SUCCINCT-NEXT: Testing:
 # SUCCINCT-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
-# SUCCINCT-NEXT: Testing:  0.. 10..
+# SUCCINCT-NEXT: Testing:
 # SUCCINCT-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
-# SUCCINCT-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# SUCCINCT-NEXT: Testing:
 # SUCCINCT-NEXT: ********************
 # SUCCINCT-NEXT: Failed Tests (1):
 # SUCCINCT-NEXT:   verbosity :: fail.txt
@@ -561,15 +561,15 @@
 # PROGRESS:      -- Testing: 5 tests, 1 workers --
 # PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
-# PROGRESS-NEXT: Testing:  0..
+# PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: PASS: verbosity :: pass.txt (2 of 5)
-# PROGRESS-NEXT: Testing:  0.. 10..
+# PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
-# PROGRESS-NEXT: Testing:  0.. 10.. 20..
+# PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
-# PROGRESS-NEXT: Testing:  0.. 10.. 20.. 30..
+# PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
-# PROGRESS-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# PROGRESS-NEXT: Testing:
 # PROGRESS-NEXT: ********************
 # PROGRESS-NEXT: Failed Tests (1):
 # PROGRESS-NEXT:   verbosity :: fail.txt
@@ -643,7 +643,7 @@
 # AS-NEXT: --
 # AS-EMPTY:
 # AS-NEXT: ********************
-# AS-NEXT: Testing:  0.. 10..
+# AS-NEXT: Testing:
 # AS-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
 # AS-NEXT: ******************** TEST 'verbosity :: xpass.txt' FAILED ********************
 # AS-NEXT: Exit Code: 0
@@ -660,7 +660,7 @@
 # AS-NEXT: --
 # AS-EMPTY:
 # AS-NEXT: ********************
-# AS-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# AS-NEXT: Testing:
 # AS-NEXT: ********************
 # AS-NEXT: Failed Tests (1):
 # AS-NEXT:   verbosity :: fail.txt
@@ -709,7 +709,7 @@
 # SA-NEXT: --
 # SA-EMPTY:
 # SA-NEXT: ********************
-# SA-NEXT: Testing:  0.. 10..
+# SA-NEXT: Testing:
 # SA-NEXT: PASS: verbosity :: pass.txt (2 of 5)
 # SA-NEXT: Exit Code: 0
 # SA-EMPTY:
@@ -725,11 +725,11 @@
 # SA-NEXT: --
 # SA-EMPTY:
 # SA-NEXT: ********************
-# SA-NEXT: Testing:  0.. 10.. 20..
+# SA-NEXT: Testing:
 # SA-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
 # SA-NEXT: Test requires the following unavailable features: asdf
 # SA-NEXT: ********************
-# SA-NEXT: Testing:  0.. 10.. 20.. 30..
+# SA-NEXT: Testing:
 # SA-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
 # SA-NEXT: Exit Code: 1
 # SA-EMPTY:
@@ -746,7 +746,7 @@
 # SA-NEXT: --
 # SA-EMPTY:
 # SA-NEXT: ********************
-# SA-NEXT: Testing:  0.. 10.. 20.. 30.. 40..
+# SA-NEXT: Testing:
 # SA-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
 # SA-NEXT: ******************** TEST 'verbosity :: xpass.txt' FAILED ********************
 # SA-NEXT: Exit Code: 0
@@ -763,7 +763,7 @@
 # SA-NEXT: --
 # SA-EMPTY:
 # SA-NEXT: ********************
-# SA-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# SA-NEXT: Testing:
 # SA-NEXT: ********************
 # SA-NEXT: Failed Tests (1):
 # SA-NEXT:   verbosity :: fail.txt
@@ -907,13 +907,13 @@
 # SQAV-NEXT: --
 # SQAV-EMPTY:
 # SQAV-NEXT: ********************
-# SQAV-NEXT: Testing:  0.. 10..
+# SQAV-NEXT: Testing:
 # SQAV-NEXT: PASS: verbosity :: pass.txt (2 of 5)
-# SQAV-NEXT: Testing:  0.. 10.. 20..
+# SQAV-NEXT: Testing:
 # SQAV-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
-# SQAV-NEXT: Testing:  0.. 10.. 20.. 30..
+# SQAV-NEXT: Testing:
 # SQAV-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
-# SQAV-NEXT: Testing:  0.. 10.. 20.. 30.. 40..
+# SQAV-NEXT: Testing:
 # SQAV-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
 # SQAV-NEXT: ******************** TEST 'verbosity :: xpass.txt' FAILED ********************
 # SQAV-NEXT: Exit Code: 0
@@ -930,7 +930,7 @@
 # SQAV-NEXT: --
 # SQAV-EMPTY:
 # SQAV-NEXT: ********************
-# SQAV-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# SQAV-NEXT: Testing:
 # SQAV-NEXT: ********************
 # SQAV-NEXT: Failed Tests (1):
 # SQAV-NEXT:   verbosity :: fail.txt
@@ -980,9 +980,9 @@
 # QUIET-W-PROGRESS: -- Testing: 5 tests, 1 workers --
 # QUIET-W-PROGRESS-NEXT: Testing:
 # QUIET-W-PROGRESS-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
-# QUIET-W-PROGRESS-NEXT: Testing:  0.. 10..
+# QUIET-W-PROGRESS-NEXT: Testing:
 # QUIET-W-PROGRESS-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
-# QUIET-W-PROGRESS-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# QUIET-W-PROGRESS-NEXT: Testing:
 # QUIET-W-PROGRESS-NEXT: ********************
 # QUIET-W-PROGRESS-NEXT: Failed Tests (1):
 # QUIET-W-PROGRESS-NEXT:   verbosity :: fail.txt
@@ -1102,15 +1102,15 @@
 # SUCCINCT-RESULT-ALL:      -- Testing: 5 tests, 1 workers --
 # SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: FAIL: verbosity :: fail.txt (1 of 5)
-# SUCCINCT-RESULT-ALL-NEXT: Testing:  0.. 10.
+# SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: PASS: verbosity :: pass.txt (2 of 5)
-# SUCCINCT-RESULT-ALL-NEXT: Testing:  0.. 10.. 20..
+# SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: {{UN}}SUPPORTED: verbosity :: unsupported.txt (3 of 5)
-# SUCCINCT-RESULT-ALL-NEXT: Testing:  0.. 10.. 20.. 30..
+# SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: {{X}}FAIL: verbosity :: xfail.txt (4 of 5)
-# SUCCINCT-RESULT-ALL-NEXT: Testing:  0.. 10.. 20.. 30.. 40..
+# SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: XPASS: verbosity :: xpass.txt (5 of 5)
-# SUCCINCT-RESULT-ALL-NEXT: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
+# SUCCINCT-RESULT-ALL-NEXT: Testing:
 # SUCCINCT-RESULT-ALL-NEXT: ********************
 # SUCCINCT-RESULT-ALL-NEXT: Failed Tests (1):
 # SUCCINCT-RESULT-ALL-NEXT:   verbosity :: fail.txt

dyung

LGTM. Thanks for fixing!

llvm-ci · 2025-11-13T02:17:22Z

LLVM Buildbot has detected a new failure on builder sanitizer-x86_64-linux-android running on sanitizer-buildbot-android while building llvm at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/186/builds/13923

Here is the relevant piece of the build log for the reference

Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure)
...
[       OK ] AddressSanitizer.AtoiAndFriendsOOBTest (2262 ms)
[ RUN      ] AddressSanitizer.HasFeatureAddressSanitizerTest
[       OK ] AddressSanitizer.HasFeatureAddressSanitizerTest (0 ms)
[ RUN      ] AddressSanitizer.CallocReturnsZeroMem
[       OK ] AddressSanitizer.CallocReturnsZeroMem (19 ms)
[ DISABLED ] AddressSanitizer.DISABLED_TSDTest
[ RUN      ] AddressSanitizer.IgnoreTest
[       OK ] AddressSanitizer.IgnoreTest (0 ms)
[ RUN      ] AddressSanitizer.SignalTest
[       OK ] AddressSanitizer.SignalTest (187 ms)
[ RUN      ] AddressSanitizer.ReallocTest
[       OK ] AddressSanitizer.ReallocTest (35 ms)
[ RUN      ] AddressSanitizer.WrongFreeTest
[       OK ] AddressSanitizer.WrongFreeTest (139 ms)
[ RUN      ] AddressSanitizer.LongJmpTest
[       OK ] AddressSanitizer.LongJmpTest (0 ms)
[ RUN      ] AddressSanitizer.ThreadStackReuseTest
[       OK ] AddressSanitizer.ThreadStackReuseTest (10 ms)
[ DISABLED ] AddressSanitizer.DISABLED_MemIntrinsicUnalignedAccessTest
[ DISABLED ] AddressSanitizer.DISABLED_LargeFunctionSymbolizeTest
[ DISABLED ] AddressSanitizer.DISABLED_MallocFreeUnwindAndSymbolizeTest
[ RUN      ] AddressSanitizer.UseThenFreeThenUseTest
[       OK ] AddressSanitizer.UseThenFreeThenUseTest (125 ms)
[ RUN      ] AddressSanitizer.FileNameInGlobalReportTest
[       OK ] AddressSanitizer.FileNameInGlobalReportTest (121 ms)
[ DISABLED ] AddressSanitizer.DISABLED_StressStackReuseAndExceptionsTest
[ RUN      ] AddressSanitizer.MlockTest
[       OK ] AddressSanitizer.MlockTest (0 ms)
[ DISABLED ] AddressSanitizer.DISABLED_DemoThreadedTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoStackTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoThreadStackTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowIn
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowLeft
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowRight
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFHigh
[ DISABLED ] AddressSanitizer.DISABLED_DemoOOM
[ DISABLED ] AddressSanitizer.DISABLED_DemoDoubleFreeTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoNullDerefTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoFunctionStaticTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoTooMuchMemoryTest
[ RUN      ] AddressSanitizer.LongDoubleNegativeTest
[       OK ] AddressSanitizer.LongDoubleNegativeTest (0 ms)
[----------] 19 tests from AddressSanitizer (27886 ms total)

[----------] Global test environment tear-down
[==========] 22 tests from 2 test suites ran. (27904 ms total)
[  PASSED  ] 22 tests.

  YOU HAVE 1 DISABLED TEST

Step 34 (run instrumented asan tests [aarch64/bluejay-userdebug/TQ3A.230805.001]) failure: run instrumented asan tests [aarch64/bluejay-userdebug/TQ3A.230805.001] (failure)
...
[ RUN      ] AddressSanitizer.HasFeatureAddressSanitizerTest
[       OK ] AddressSanitizer.HasFeatureAddressSanitizerTest (0 ms)
[ RUN      ] AddressSanitizer.CallocReturnsZeroMem
[       OK ] AddressSanitizer.CallocReturnsZeroMem (19 ms)
[ DISABLED ] AddressSanitizer.DISABLED_TSDTest
[ RUN      ] AddressSanitizer.IgnoreTest
[       OK ] AddressSanitizer.IgnoreTest (0 ms)
[ RUN      ] AddressSanitizer.SignalTest
[       OK ] AddressSanitizer.SignalTest (187 ms)
[ RUN      ] AddressSanitizer.ReallocTest
[       OK ] AddressSanitizer.ReallocTest (35 ms)
[ RUN      ] AddressSanitizer.WrongFreeTest
[       OK ] AddressSanitizer.WrongFreeTest (139 ms)
[ RUN      ] AddressSanitizer.LongJmpTest
[       OK ] AddressSanitizer.LongJmpTest (0 ms)
[ RUN      ] AddressSanitizer.ThreadStackReuseTest
[       OK ] AddressSanitizer.ThreadStackReuseTest (10 ms)
[ DISABLED ] AddressSanitizer.DISABLED_MemIntrinsicUnalignedAccessTest
[ DISABLED ] AddressSanitizer.DISABLED_LargeFunctionSymbolizeTest
[ DISABLED ] AddressSanitizer.DISABLED_MallocFreeUnwindAndSymbolizeTest
[ RUN      ] AddressSanitizer.UseThenFreeThenUseTest
[       OK ] AddressSanitizer.UseThenFreeThenUseTest (125 ms)
[ RUN      ] AddressSanitizer.FileNameInGlobalReportTest
[       OK ] AddressSanitizer.FileNameInGlobalReportTest (121 ms)
[ DISABLED ] AddressSanitizer.DISABLED_StressStackReuseAndExceptionsTest
[ RUN      ] AddressSanitizer.MlockTest
[       OK ] AddressSanitizer.MlockTest (0 ms)
[ DISABLED ] AddressSanitizer.DISABLED_DemoThreadedTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoStackTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoThreadStackTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowIn
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowLeft
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFLowRight
[ DISABLED ] AddressSanitizer.DISABLED_DemoUAFHigh
[ DISABLED ] AddressSanitizer.DISABLED_DemoOOM
[ DISABLED ] AddressSanitizer.DISABLED_DemoDoubleFreeTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoNullDerefTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoFunctionStaticTest
[ DISABLED ] AddressSanitizer.DISABLED_DemoTooMuchMemoryTest
[ RUN      ] AddressSanitizer.LongDoubleNegativeTest
[       OK ] AddressSanitizer.LongDoubleNegativeTest (0 ms)
[----------] 19 tests from AddressSanitizer (27886 ms total)

[----------] Global test environment tear-down
[==========] 22 tests from 2 test suites ran. (27904 ms total)
[  PASSED  ] 22 tests.

  YOU HAVE 1 DISABLED TEST
program finished with exit code 0
elapsedTime=2337.722398

hnrklssn · 2025-11-13T02:47:20Z

LLVM Buildbot has detected a new failure on builder sanitizer-x86_64-linux-android running on sanitizer-buildbot-android while building llvm at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/186/builds/13923
Here is the relevant piece of the build log for the reference

I can't seem to figure out what actually failed here?

The rate at which lit's progress bar progresses is not deterministic. Don't try to match the contents of the progress bar, only the fact that it's there.

rorth · 2025-11-13T10:38:33Z

This patch broke the Solaris/amd64 buildbot (and likely also will the Solaris/sparcv9 one once it catches up).

I suspect the test is confused by the

warning: Failed to raise process limit: module 'resource' has no attribute 'RLIMIT_NPROC'

warning.

rorth · 2025-11-13T16:02:56Z

Indeed the Solaris/sparcv9 bot is affected just the same.

hnrklssn · 2025-11-13T16:10:30Z

Indeed the Solaris/sparcv9 bot is affected just the same.

I can fix in a moment when I'm at a computer.

hnrklssn · 2025-11-13T17:52:30Z

Solaris failure should be fixed here: #167921

[utils] remove flakiness in verbosity.py test case

de43a76

The rate at which lit's progress bar progresses is not deterministic. Don't try to match the contents of the progress bar, only the fact that it's there.

hnrklssn requested a review from dyung November 13, 2025 01:04

llvmbot added llvm-lit testing-tools labels Nov 13, 2025

hnrklssn enabled auto-merge (squash) November 13, 2025 01:05

hnrklssn mentioned this pull request Nov 13, 2025

[utils] revamp options controlling lit's output #167192

Merged

dyung approved these changes Nov 13, 2025

View reviewed changes

hnrklssn merged commit 6121826 into llvm:main Nov 13, 2025
12 of 13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[utils] remove flakiness in verbosity.py test case #167801

[utils] remove flakiness in verbosity.py test case #167801

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

llvmbot commented Nov 13, 2025

Uh oh!

dyung left a comment

Uh oh!

Uh oh!

llvm-ci commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

rorth commented Nov 13, 2025

Uh oh!

rorth commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[utils] remove flakiness in verbosity.py test case #167801

[utils] remove flakiness in verbosity.py test case #167801

Uh oh!

Conversation

hnrklssn commented Nov 13, 2025

Uh oh!

llvmbot commented Nov 13, 2025

Uh oh!

dyung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

llvm-ci commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

rorth commented Nov 13, 2025

Uh oh!

rorth commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

hnrklssn commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants