[SYCL][Unit Tests] Fix regression after https://github.com/intel/llvm/pull/19687 #20534

dm-vodopyanov · 2025-11-03T15:42:57Z

This patch fixes a regression happened after #19687:
check-all lost the entire SYCL-Unit test suite, so SYCL unit tests did not run for check-all target.

This patch fixes a regression happened after intel#19687: `check-all` lost the entire SYCL-Unit test suite, so SYCL unit tests did not run for `check-all` target.

dm-vodopyanov · 2025-11-03T15:49:30Z

sycl/test/Unit/lit.cfg.py


 # testFormat: The test format to use to interpret tests.
-config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, "Tests")
+config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, test_suffix="Tests_non_preview")


Not sure what is better:
Option 1: add a workaroud here as it is now: we call the existing ctor, and when modify attribute self.test_suffixes
Option 2: modify llvm/llvm/utils/lit/lit/formats/googletest.py but it will be a modification of common LLVM sources, so if community changes something there, it will break on our end. Modification means add a ctor which takes list of test suffixes. I think suggesting the patch to the community is not an option as we need to fix this very fast.

I think I prefer a combination of the two: Go with Option 1 for now, then open a change to upstream LLVM with the modifications in Option 2. When it is merged upstream and it reaches intel/llvm we can drop the workaround.

Also Option 3 exists: refactor our CMake files so that even for non_preview the test binary name suffix is still Tests

As pre-commit shows, it looks like it is better to separate non_preview and preview tests by dividing SYCL-Unit into SYCL-Unit-Preview and SYCL-Unit-Non-Preview as the preview and non-preview variations of the same test can run simultaneously and we can get a data race.

In c8999d6: kept single test_prefix (Tests). Test names started to look ugly, like SchedulerTests_Non_Preview_Tests, but I think it could be temporary. Also disabled tests which used the same external recourses and as a result failed due to data races. These tests should be re-written. I will create a ticket against SYCL RT team and add the ticket number to the test sources.

Rather than completely disabling unit tests for testing preview breaking changes, you could do:

config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, "Non_Preview_Tests")

in sycl/test/Unit/lit.cfg.py - so that we run only non-preview unitests as a part of check-sycl/check-all. This won't reduce testing coverage as we'd still be running preview version of SYCL unittests in intel/llvm CI as part of ninja check-sycl-unittests

@uditagarwal97 But what about our feature board? Let's imagine the situation that we need to see the results with preview version of library in it. In that case they won't be there.

@uditagarwal97 But what about our feature board? Let's imagine the situation that we need to see the results with preview version of library in it. In that case they won't be there.

Why would we need such results? Either we can't break ABI meaning that our main goal is to cover regular path with tests, or we can break ABI in which case let's just break it and just have one code path instead of two

@uditagarwal97 But what about our feature board? Let's imagine the situation that we need to see the results with preview version of library in it. In that case they won't be there.

Why would we need such results? Either we can't break ABI meaning that our main goal is to cover regular path with tests, or we can break ABI in which case let's just break it and just have one code path instead of two

OK. Let's include only non-preview tests to check-all as part of this specific PR.

Ready to review.

sycl/cmake/modules/AddSYCLUnitTest.cmake

uditagarwal97 · 2025-11-04T16:53:40Z

sycl/unittests/config/ConfigTests.cpp

 #include <regex>

-TEST(ConfigTests, CheckConfigProcessing) {
+TEST(ConfigTests, DISABLED_CheckConfigProcessing) {


Are all the disabled unittests failing because they set some environment variable which gets unintentionally modified when we run preview and non-preview unittests parallelly?

IIRC, preview and non-preview unit tests are in different folders so there shouldn't be any race on filesystem...
Btw, in CI, when we run check-sycl-unitests, we run both preview and non-preview unittests and we don't see any failure there.

What's the difference in our setup when we run unit tests via check-sycl-unitests and check-sycl? It's weird that unitests are failing with one but not another.

I think that the main difference between check-sycl-unittests and check-sycl is that check-sycl-unittests handles different tests at build system level, i.e. each test executable is a separate target which are all run sequentially:

https://github.com/intel/llvm/blob/e05e82b240afe298faca4f89ec448772e4123b12/sycl/cmake/modules/AddSYCLUnitTest.cmake#L104C42-L104C63

From CI logs:

cmake --build $GITHUB_WORKSPACE/build --target check-sycl-unittests

I don't think that cmake runs more than 1 thread by default.

check-sycl on the other hand is defined as a LIT test suite:

llvm/sycl/test/CMakeLists.txt

Lines 79 to 85 in e05e82b

add_lit_testsuite(check-sycl-combined-triples "Running device-agnostic SYCL regression tests for all available triples"

${CMAKE_CURRENT_BINARY_DIR}

ARGS ${RT_TEST_ARGS}

PARAMS "SYCL_TRIPLE=${TRIPLES}"

DEPENDS ${SYCL_TEST_DEPS}

${SYCL_TEST_EXCLUDE}

)

And therefore it is by default launched in highly parallel mode by LIT where each TEST is a separate LIT test and can be run in parallel with other tests.

So yes, our CI simply runs everything sequentially in one thread, hence no races. LIT runs everything in parallel - and I assume that working directory for all tests is the same, so conf.txt file that we create is shared by more then one tests (i.e. its preview and non-preview versions)

Makes a lot of sense. Thanks!

sycl/cmake/modules/AddSYCLUnitTest.cmake

github-actions · 2025-11-06T13:13:47Z

@intel/llvm-gatekeepers please consider merging

[SYCL][Unit Tests] Fix regression after intel#19687

194e20b

This patch fixes a regression happened after intel#19687: `check-all` lost the entire SYCL-Unit test suite, so SYCL unit tests did not run for `check-all` target.

dm-vodopyanov requested a review from a team as a code owner November 3, 2025 15:42

dm-vodopyanov requested a review from steffenlarsen November 3, 2025 15:42

dm-vodopyanov had a problem deploying to WindowsCILock November 3, 2025 15:43 — with GitHub Actions Error

dm-vodopyanov commented Nov 3, 2025

View reviewed changes

Fix clang format

360c60e

dm-vodopyanov had a problem deploying to WindowsCILock November 3, 2025 15:56 — with GitHub Actions Failure

dm-vodopyanov temporarily deployed to WindowsCILock November 3, 2025 16:25 — with GitHub Actions Inactive

dm-vodopyanov requested a review from uditagarwal97 November 4, 2025 14:22

Update

c8999d6

dm-vodopyanov temporarily deployed to WindowsCILock November 4, 2025 16:18 — with GitHub Actions Inactive

uditagarwal97 reviewed Nov 4, 2025

View reviewed changes

sycl/cmake/modules/AddSYCLUnitTest.cmake Show resolved Hide resolved

dm-vodopyanov had a problem deploying to WindowsCILock November 4, 2025 16:53 — with GitHub Actions Failure

dm-vodopyanov temporarily deployed to WindowsCILock November 4, 2025 16:53 — with GitHub Actions Inactive

uditagarwal97 reviewed Nov 4, 2025

View reviewed changes

Update

8d543d7

dm-vodopyanov had a problem deploying to WindowsCILock November 5, 2025 15:06 — with GitHub Actions Error

Update

69b4775

dm-vodopyanov temporarily deployed to WindowsCILock November 5, 2025 15:09 — with GitHub Actions Inactive

dm-vodopyanov temporarily deployed to WindowsCILock November 5, 2025 15:35 — with GitHub Actions Inactive

dm-vodopyanov had a problem deploying to WindowsCILock November 5, 2025 15:35 — with GitHub Actions Error

uditagarwal97 approved these changes Nov 5, 2025

View reviewed changes

uditagarwal97 reviewed Nov 5, 2025

View reviewed changes

sycl/cmake/modules/AddSYCLUnitTest.cmake Show resolved Hide resolved

Update

02ae68a

dm-vodopyanov temporarily deployed to WindowsCILock November 5, 2025 16:30 — with GitHub Actions Inactive

dm-vodopyanov temporarily deployed to WindowsCILock November 5, 2025 17:00 — with GitHub Actions Inactive

dm-vodopyanov merged commit 0fa2cd4 into intel:sycl Nov 6, 2025
44 of 45 checks passed

	add_lit_testsuite(check-sycl-combined-triples "Running device-agnostic SYCL regression tests for all available triples"
	${CMAKE_CURRENT_BINARY_DIR}
	ARGS ${RT_TEST_ARGS}
	PARAMS "SYCL_TRIPLE=${TRIPLES}"
	DEPENDS ${SYCL_TEST_DEPS}
	${SYCL_TEST_EXCLUDE}
	)

[SYCL][Unit Tests] Fix regression after https://github.com/intel/llvm/pull/19687 #20534

[SYCL][Unit Tests] Fix regression after https://github.com/intel/llvm/pull/19687 #20534

Uh oh!

Conversation

dm-vodopyanov commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dm-vodopyanov Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Nov 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dm-vodopyanov commented Nov 3, 2025 •

edited

Loading

dm-vodopyanov Nov 3, 2025 •

edited

Loading