[lldb][test] Don't run libc++ API tests without a locally built libc++ #162657

Michael137 · 2025-10-09T13:38:56Z

API tests in the libc++ category will try their best to build against a locally built libc++. If none exists, the Makefile.rules currently fall back to using the system libc++.

The issue with falling back to the system libc++ is that we are now potentially not testing what we intended to. But we also can't rely on certain libc++ features being available that the tests are trying to use. On Apple platforms this is a configuration error (because libc++ is the only stdlib supported), but we can't make it an error on Linux because a user might want to run the API tests with libstdc++.

The Ubunutu 22.04 bots on the Apple fork are failing to run following tests are failing:

TestLibcxxInternalsRecognizer.py
TestDataFormatterStdRangesRefView.py because the system stdlib doesn't have std::ranges support yet. And the tests just fail to build. Building libc++ on those bots is also not possible because the system compiler is too old (and the Apple fork builds all the subprojects standalone, so it requires the system compiler).

This patch marks tests in the libc++ category as UNSUPPORTED if no local libc++ is available.

The downside is that we will inevitably lose coverage on bots that were running these tests without a local libc++. Arguably those weren't really testing the right thing. But for vendors with LLDB forks it might have been useful to at least know that the tests on the fork don't fail against the system libc++.

Confirmed that the libc++ pre-merge CI still runs these tests (since it uses the explicit --category libc++ dotest flag). Also confirmed that LLDB pre-merge CI runs the tests (because it builds libcxx locally).

Workaround

If you do need want to run libc++ tests against the system stdlib, you can invoke lldb-dotest with the --category libc++ flag:

./path/to/build/lldb-dotest --category libc++

OR

./path/to/build/bin/llvm-lit -sv --param dotest-args='--category libc++' "/path/to/monorepo/lldb/test/API

rdar://136231390

API tests in the `libc++` category will try their best to build against a locally built libc++. If none exists, the `Makefile.rules` currently fall back to using the system libc++. The issue with falling back to the system libc++ is that we are now potentially not testing what we intended to. But we also can't rely on certain libc++ features being available that the tests are trying to use. On Apple platforms this is a configuration error (because libc++ is the only stdlib supported), but we can't make it an error on Linux because a user might want to run the API tests with libstdc++. The Ubunutu 22.04 bots on the Apple fork are failing to run following tests are failing: * `TestLibcxxInternalsRecognizer.py` * `TestDataFormatterStdRangesRefView.py` because the system stdlib doesn't have `std::ranges` support yet. And the tests just fail to build. Building libc++ on those bots is also not possible because the system compiler is too old (and the Apple fork builds all the subprojects standalone, so it requires the system compiler). This patch marks tests in the `libc++` category as `UNSUPPORTED` if no local libc++ is available. The downside is that we will inevitably lose coverage on bots that were running these tests without a local libc++. Arguably those weren't really testing the right thing. But for vendors with LLDB forks it might have been useful to at least know that the tests on the fork don't fail against the system libc++.

github-actions · 2025-10-09T14:11:00Z

✅ With the latest revision this PR passed the C/C++ code formatter.

llvmbot · 2025-10-09T16:27:19Z

@llvm/pr-subscribers-lldb

Author: Michael Buch (Michael137)

Changes

API tests in the libc++ category will try their best to build against a locally built libc++. If none exists, the Makefile.rules currently fall back to using the system libc++.

The issue with falling back to the system libc++ is that we are now potentially not testing what we intended to. But we also can't rely on certain libc++ features being available that the tests are trying to use. On Apple platforms this is a configuration error (because libc++ is the only stdlib supported), but we can't make it an error on Linux because a user might want to run the API tests with libstdc++.

The Ubunutu 22.04 bots on the Apple fork are failing to run following tests are failing:

TestLibcxxInternalsRecognizer.py
TestDataFormatterStdRangesRefView.py because the system stdlib doesn't have std::ranges support yet. And the tests just fail to build. Building libc++ on those bots is also not possible because the system compiler is too old (and the Apple fork builds all the subprojects standalone, so it requires the system compiler).

This patch marks tests in the libc++ category as UNSUPPORTED if no local libc++ is available.

The downside is that we will inevitably lose coverage on bots that were running these tests without a local libc++. Arguably those weren't really testing the right thing. But for vendors with LLDB forks it might have been useful to at least know that the tests on the fork don't fail against the system libc++.

rdar://136231390

Full diff: https://github.com/llvm/llvm-project/pull/162657.diff

1 Files Affected:

(modified) lldb/packages/Python/lldbsuite/test/dotest.py (+4)

diff --git a/lldb/packages/Python/lldbsuite/test/dotest.py b/lldb/packages/Python/lldbsuite/test/dotest.py
index 2966ac04227cb..1d474a2180d22 100644
--- a/lldb/packages/Python/lldbsuite/test/dotest.py
+++ b/lldb/packages/Python/lldbsuite/test/dotest.py
@@ -789,6 +789,10 @@ def canRunLibcxxTests():
         return True, "libc++ always present"
 
     if platform == "linux":
+        if not configuration.libcxx_include_dir or not configuration.libcxx_library_dir:
+            return False, "API tests require a locally built libc++."
+
+        # Make sure -stdlib=libc++ works since that's how the tests will be built.
         with temp_file.OnDiskTempFile() as f:
             cmd = [configuration.compiler, "-xc++", "-stdlib=libc++", "-o", f.path, "-"]
             p = subprocess.Popen(

adrian-prantl · 2025-10-09T16:38:46Z

Were there tests that are in that category that were successfully testing the Python libstdc++ data formatters that would now be no longer run?

Michael137 · 2025-10-10T06:49:21Z

Were there tests that are in that category that were successfully testing the Python libstdc++ data formatters that would now be no longer run?

Nope those should be all under the libstdc++ test category

DavidSpickett · 2025-10-10T12:11:18Z

Just to confirm, Linaro's bots were not running these tests. Not losing anything there.

They do run the layout simulator tests though, these will continue to run I assume?

I agree that having a way to opt-in with the category option is better than silently testing out of date libraries. Someone doing a standalone build might also use it.

lldb/packages/Python/lldbsuite/test/dotest.py

Michael137 · 2025-10-10T12:43:29Z

Just to confirm, Linaro's bots were not running these tests. Not losing anything there.

Thanks for confirming!

They do run the layout simulator tests though, these will continue to run I assume?

Yea they are not part of the libc++ category. So they will still be run:

$ ./bin/lldb-dotest --category libc++ lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx-simulators

...

Ran 68 tests in 0.012s

OK (skipped=68)

DavidSpickett · 2025-10-10T13:40:14Z

--category libc++

It shows 68 skipped, did you paste the wrong output?

Michael137 · 2025-10-10T13:41:21Z

--category libc++

It shows 68 skipped, did you paste the wrong output?

As in, I wanted to show that the simluators aren't part of the libc++ category :p

DavidSpickett · 2025-10-10T13:41:52Z

Also please add to the PR description a single line like:

If you need to re-enable these tests do ....

In case the reader is a person who cared about system library compatibility.

You kinda say what to do already but not in a Friday afternoon in a rush compatible way :)

Michael137 · 2025-10-10T13:48:18Z

Also please add to the PR description a single line like:
If you need to re-enable these tests do ....
In case the reader is a person who cared about system library compatibility.

You kinda say what to do already but not in a Friday afternoon in a rush compatible way :)

done! let me know if you want me to reword it

DavidSpickett · 2025-10-10T14:00:57Z

As in, I wanted to show that the simluators aren't part of the libc++ category :p

Ok I see now. You enabled only that category, to show that no simulator test falls into it. Rather than disabling the category, to show that all the simulator tests ran (because how would I know exactly how many there are).

👍

DavidSpickett

LGTM.

da-viper · 2025-10-10T14:22:35Z

To confirm that means that, the libc++ tests are no longer tested against the system's libc++ and to enable libc++ test you have to add libcxx to LLVM_ENABLE_RUNTIMES ?

Michael137 · 2025-10-10T14:41:17Z

To confirm that means that, the libc++ tests are no longer tested against the system's libc++ and to enable libc++ test you have to add libcxx to LLVM_ENABLE_RUNTIMES ?

Correct

llvm#162657) API tests in the `libc++` category will try their best to build against a locally built libc++. If none exists, the `Makefile.rules` currently fall back to using the system libc++. The issue with falling back to the system libc++ is that we are now potentially not testing what we intended to. But we also can't rely on certain libc++ features being available that the tests are trying to use. On Apple platforms this is a configuration error (because libc++ is the only stdlib supported), but we can't make it an error on Linux because a user might want to run the API tests with libstdc++. The Ubunutu 22.04 bots on the Apple fork are failing to run following tests are failing: * `TestLibcxxInternalsRecognizer.py` * `TestDataFormatterStdRangesRefView.py` because the system stdlib doesn't have `std::ranges` support yet. And the tests just fail to build. Building libc++ on those bots is also not possible because the system compiler is too old (and the Apple fork builds all the subprojects standalone, so it requires the system compiler). This patch marks tests in the `libc++` category as `UNSUPPORTED` if no local libc++ is available. The downside is that we will inevitably lose coverage on bots that were running these tests without a local libc++. Arguably those weren't really testing the right thing. But for vendors with LLDB forks it might have been useful to at least know that the tests on the fork don't fail against the system libc++. Confirmed that the libc++ pre-merge CI still runs these tests (since it uses the explicit `--category libc++` dotest flag). Also confirmed that LLDB pre-merge CI runs the tests (because it builds `libcxx` locally). **Workaround** If you do need want to run libc++ tests against the system stdlib, you can invoke `lldb-dotest` with the `--category libc++` flag: ``` ./path/to/build/lldb-dotest --category libc++ OR ./path/to/build/bin/llvm-lit -sv --param dotest-args='--category libc++' "/path/to/monorepo/lldb/test/API ``` rdar://136231390 (cherry picked from commit 4188e18)

Michael137 requested review from DavidSpickett, adrian-prantl and labath October 9, 2025 13:38

fixup! move comment

cc9b3e8

Michael137 force-pushed the lldb/libcxx-test branch from 2ed5ef0 to cc9b3e8 Compare October 9, 2025 16:25

Michael137 marked this pull request as ready for review October 9, 2025 16:26

Michael137 requested a review from JDevlieghere as a code owner October 9, 2025 16:26

llvmbot added the lldb label Oct 9, 2025

adrian-prantl requested a review from da-viper October 9, 2025 16:38

DavidSpickett reviewed Oct 10, 2025

View reviewed changes

lldb/packages/Python/lldbsuite/test/dotest.py Show resolved Hide resolved

DavidSpickett approved these changes Oct 10, 2025

View reviewed changes

adrian-prantl approved these changes Oct 10, 2025

View reviewed changes

Michael137 merged commit 4188e18 into llvm:main Oct 10, 2025
12 checks passed

Michael137 deleted the lldb/libcxx-test branch October 10, 2025 18:13

[lldb][test] Don't run libc++ API tests without a locally built libc++ #162657

[lldb][test] Don't run libc++ API tests without a locally built libc++ #162657

Uh oh!

Conversation

Michael137 commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Oct 9, 2025

Uh oh!

adrian-prantl commented Oct 9, 2025

Uh oh!

Michael137 commented Oct 10, 2025

Uh oh!

DavidSpickett commented Oct 10, 2025

Uh oh!

Uh oh!

Michael137 commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidSpickett commented Oct 10, 2025

Uh oh!

Michael137 commented Oct 10, 2025

Uh oh!

DavidSpickett commented Oct 10, 2025

Uh oh!

Michael137 commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidSpickett commented Oct 10, 2025

Uh oh!

DavidSpickett left a comment

Choose a reason for hiding this comment

Uh oh!

da-viper commented Oct 10, 2025

Uh oh!

Michael137 commented Oct 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Michael137 commented Oct 9, 2025 •

edited

Loading

github-actions bot commented Oct 9, 2025 •

edited

Loading

Michael137 commented Oct 10, 2025 •

edited

Loading

Michael137 commented Oct 10, 2025 •

edited

Loading