runtests: add options to set minimum number of tests, use them #19942
Conversation
vszakats
changed the title
There was a problem hiding this comment.
Pull request overview
This PR adds functionality to enforce a minimum number of tests to run in the test suite, helping detect CI configurations that inadvertently skip most tests. The feature can be set via the --min command-line option or the CURL_TEST_MIN environment variable, with the command-line option taking precedence.
Key changes:
- Adds
--min=[count]option to runtests.pl for setting minimum test count - Implements validation logic that fails the test run if fewer than the minimum tests are executed
- Sets global minimum of 1600 tests for Linux CI and specific minimums for various test configurations
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| tests/runtests.pl | Implements --min argument parsing and validation logic to enforce minimum test counts |
| tests/globalconfig.pm | Declares and exports the $mintotal global variable for tracking minimum test requirements |
| docs/runtests.md | Documents the new -min option and its relationship to the CURL_TEST_MIN environment variable |
| .github/workflows/linux.yml | Sets default minimum test count (1600) and configures specific minimums for individual test jobs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
vszakats
force-pushed
the
minruntests
branch
from
5a16cbe to
1ba11bf
Compare
|
(since yesterday's GitHub update, |
vszakats
changed the title
In my opinion that is a bug in the beta test. I have reported it here. Check if you have beta / "new experience" / preview enabled: 
|
|
@jay Phew, thanks for the tip and reporting it. I did have it enabled. Turned it off |
Ref: curl#19987 Follow-up to 3f1cd80 curl#19942
Tailored for each job with a relatively tight limits. Also with no tolerance in valgrind tests: 4 of the 4 has to be run. Based on Test Clutch feature matrix which displays the minimum and actual number of tests: https://testclutch.curl.se/static/reports/feature-matrix.html Also: - runtests.pl: include total number of tests in the error message shown when the limit was not met. Assisted-by: Dan Fandrich Follow-up to 3f1cd80 #19942 Closes #20050
Before this patch CI silently ignored stunnel install failures. This pushed the failure to the test run step due to not meeting the minimum number of tests. Make the root cause obvious by failing early in this case. Reported-by: Stefan Eissing Follow-up to 3f1cd80 curl#19942
Before this patch CI silently ignored stunnel install failures. This pushed the failure to the test run step due to not meeting the minimum number of tests. Make the root cause obvious by failing early in this case. Reported-by: Stefan Eissing Follow-up to 3f1cd80 #19942 Ref: https://github.com/curl/curl/actions/runs/21245626382/job/61134101176?pr=20397 Ref: #16819 Closes #20409
3 participants
To detect mistakes made in the runtests framework that reduce
the number of test runs. Before this patch it could go undetected with
a green CI.
The minimum thresholds will need light maintenance going forward (either
bumping them periodically, or adjust if some may fell below minimums for
justified reasons). We may also make minimums tighter or looser, or more
job-specific.
Latest number of test runs for each job can be seen at Test Clutch:
https://testclutch.curl.se/static/reports/feature-matrix.html
Also:
Assisted-by: Dan Fandrich
Follow-up to f2a75a1
Follow-up to bb1391f #19510
https://github.com/curl/curl/pull/19942/files?w=1
https://github.com/curl/curl/pull/19942/changes?w=1
CURL_TEST_MINin more workflows/jobs.