From 4abee04ddc6ad30ad735e922e1b468ebc3092b2a Mon Sep 17 00:00:00 2001 From: Kendal Harland <3987220+kendalharland@users.noreply.github.com> Date: Fri, 16 Aug 2024 10:54:25 -0700 Subject: [PATCH] [lldb][test] Mark gtest cases as XFAIL if the test suite is XFAIL (#102986) When a test case inside of a gtest suite fails, we report a failure which causes the entire `ninja check-lldb` invocation to fail, even if the outer test case is marked as XFAIL - each test case result is reported as its own lit test run. This PR updates lit so it checks whether each test case's parent test suite is XFAIL before setting the status to FAIL. This is especially problematic because the failing tests can't manually be marked as XFAIL, due to https://github.com/llvm/llvm-project/issues/102264. Fixes https://github.com/llvm/llvm-project/issues/102265 ### Repro instructions 1. Modify any gtest test case to generate a failure. 2. Mark the outer lit test with XFAIL using either `--xfail-tests` flag or `LIT_XFAIL` env var. 3. Run the tests 4. Observe the lit test is XFAIL as expected, but the failed child test cases show up as separate failures. Co-authored-by: kendal (cherry picked from commit 9f89d31d5185015f8eea9c0f3d35e7ba9d353e67) --- llvm/utils/lit/lit/formats/googletest.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/llvm/utils/lit/lit/formats/googletest.py b/llvm/utils/lit/lit/formats/googletest.py index f8304cbd05453..b3af068cdf97c 100644 --- a/llvm/utils/lit/lit/formats/googletest.py +++ b/llvm/utils/lit/lit/formats/googletest.py @@ -301,7 +301,11 @@ def remove_gtest(tests): returnCode = lit.Test.SKIPPED elif "failures" in testinfo: has_failure_in_shard = True - returnCode = lit.Test.FAIL + returnCode = ( + lit.Test.XFAIL + if test.isExpectedToFail() + else lit.Test.FAIL + ) output = header for fail in testinfo["failures"]: output += fail["failure"] + "\n"