Skip to content

Break down CTest cases into each test suite#1931

Merged
Rangi42 merged 1 commit intogbdev:masterfrom
ISSOtm:ctest-monolithn't
Apr 13, 2026
Merged

Break down CTest cases into each test suite#1931
Rangi42 merged 1 commit intogbdev:masterfrom
ISSOtm:ctest-monolithn't

Conversation

@ISSOtm
Copy link
Copy Markdown
Member

@ISSOtm ISSOtm commented Apr 11, 2026

This allows parallelising the individual programs' test suites, which seems to save around two minutes and a half!

This is a medium-effort preview of #1927: this doesn't expose each test to CTest, but at least we get some useful amount of parallelism out of this. (Especially for our CI.)

@ISSOtm ISSOtm added enhancement Typically new features; lesser priority than bugs tests This affects the test suite labels Apr 11, 2026
@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from d2d0dcf to 12fd6a4 Compare April 11, 2026 18:43
@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch 2 times, most recently from 3e5ecac to 69ffaf4 Compare April 11, 2026 19:19
@ISSOtm
Copy link
Copy Markdown
Member Author

ISSOtm commented Apr 11, 2026

This seems to save a couple of minutes on Windows, but we're still in the realm of 10 minutes for testing. Though at least CTest is nice enough to give us a (wall-clock) breakdown:

1/6 Test #6: fetch-test-deps ..................   Passed    8.47 sec
2/6 Test #4: rgbgfx ...........................   Passed   29.16 sec
3/6 Test #2: rgblink ..........................   Passed   36.93 sec
4/6 Test #3: rgbfix ...........................   Passed  103.54 sec
5/6 Test #1: rgbasm ...........................   Passed  230.78 sec
6/6 Test #5: external .........................   Passed  357.14 sec

...which seems to imply that the slowest is running the external tests. For comparison, macOS (5m30s):

1/6 Test #6: fetch-test-deps ..................   Passed    3.66 sec
2/6 Test #2: rgblink ..........................   Passed   42.45 sec
3/6 Test #3: rgbfix ...........................   Passed   66.79 sec
4/6 Test #4: rgbgfx ...........................   Passed   67.17 sec
5/6 Test #1: rgbasm ...........................   Passed  179.85 sec
6/6 Test #5: external .........................   Passed  155.11 sec

@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from 69ffaf4 to 371e54d Compare April 11, 2026 19:49
@Rangi42
Copy link
Copy Markdown
Contributor

Rangi42 commented Apr 11, 2026

Each external test is independent; could they at least be broken down into free and nonfree categories?

@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from 371e54d to eb36754 Compare April 11, 2026 19:56
@ISSOtm
Copy link
Copy Markdown
Member Author

ISSOtm commented Apr 11, 2026

That sure would be nice! But the test runner currently doesn't allow splitting these two categories. And I've limited my effort in this PR to not touching the test runners at all, as a scope limiter.

@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from eb36754 to 9bfe422 Compare April 11, 2026 20:08
@ISSOtm ISSOtm requested a review from Rangi42 April 11, 2026 20:08
Comment on lines +378 to 379
ASAN_OPTIONS=detect_leaks=0 ctest --test-dir build --schedule-random --label-exclude external
cmake --install build --verbose
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're able to run ctest before cmake --install, then we don't need to be installing rgbds at all. That goes for the other OSes as well.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think installing is mostly used as a control for whether it works and in the correct locations. But, yeah, at this point we should probably just get rid of all --installs—in a follow-up PR, though.

@ISSOtm ISSOtm requested a review from Rangi42 April 12, 2026 21:42
@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from 9bfe422 to fd9d366 Compare April 12, 2026 21:45
This allows parallelising the individual programs'
test suites, which can save a few minutes!
@ISSOtm ISSOtm force-pushed the ctest-monolithn't branch from fd9d366 to 1491e05 Compare April 12, 2026 22:38
Copy link
Copy Markdown
Contributor

@Rangi42 Rangi42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were some comments to follow up on in more PRs (getting rid of unnecessary install commands in CI, using ProcessorCount instead of hard-coded 4, consistent ctest flags instead of --label-exclude external-repos being unique) but this is good enough. Thank you!

@Rangi42 Rangi42 merged commit 78281a4 into gbdev:master Apr 13, 2026
24 checks passed
@ISSOtm ISSOtm deleted the ctest-monolithn't branch April 13, 2026 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Typically new features; lesser priority than bugs tests This affects the test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants