gh-109566, regrtest: Add --fast-ci and --slow-ci options #109570
Conversation
|
I'm not sure that these 2 options are required. Maybe --fast-ci should simply be the default in regrtest. I'm not sure that we need some many ways to run tests. Why not having better default behavior, without any option? |
|
When Python is build to create a Linux Fedora package, we run the test suite with Maybe that's where we can put a line between "strict" and "not strict":
|
For example, in time budget, I would be fine with Obviously, by design, some tests can pass with |
|
So, |
Yes, but they also add more options: Later, I also plan to attempt to pass options to Python like: But this change is more complicated because of WASM/WASI buildbot workers which use a "host runner" thing and |
|
|
vstinner
force-pushed
the
regrtest_ci
branch
from
dbbdc26
to
70d1c17
Compare
vstinner
requested review from
erlend-aasland,
corona10,
ezio-melotti and
hugovk
as code owners
vstinner
force-pushed
the
regrtest_ci
branch
3 times, most recently
from
699a0dd
to
cf59c92
Compare
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
vstinner
force-pushed
the
regrtest_ci
branch
from
cf59c92
to
be4c7be
Compare
|
I removed |
vstinner
added
the
🔨 test-with-buildbots
bedevere-bot
removed
the
🔨 test-with-buildbots
Git step failed.
test_tools failed: issue gh-109615. |
|
|
|
|
Done in follow-up change: PR gh-109909. |
|
Thanks for looking into the change @gpshead :-) |
…n#109570) * Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.
| @@ -56,20 +53,13 @@ def main(regrtest_args): | |||
| args.extend(test.support.args_from_interpreter_flags()) | |||
|
|
|||
| args.extend(['-m', 'test', # Run the test suite | |||
| '-r', # Randomize test order | |||
| '-w', # Re-run failed tests in verbose mode | |||
| '--fast-ci', # Fast Continuous Integration mode | |||
There was a problem hiding this comment.
is this supposed to be here? I see it called from the Makefile with --slow-ci and --fast-ci -- but wouldn't this override either of those?
There was a problem hiding this comment.
make test runs python -E -m test --fast-ci --fast-ci --timeout=. Passing the option twice is fine.
make buildbottest runs python -E -m test --fast-ci --slow-ci --timeout=. If regrtest gets --fast-ci and --slow-ci, --slow-ci has the priority.
I would prefer to not duplicate these fast/slow CI options. I would also prefer to remove the run_tests.py script, but it's needed for cross-compiled Python.
It doesn't look too complicated to move run_tests.py code into libregrtest.
There was a problem hiding this comment.
makes sense -- just double checking since I got a weird conflict here (debian/deadsnakes patches out the randomization because it makes the builds nondeterministic)
There was a problem hiding this comment.
I created issue gh-110152 for that.
There was a problem hiding this comment.
makes sense -- just double checking since I got a weird conflict here (debian/deadsnakes patches out the randomization because it makes the builds nondeterministic)
Do you set SOURCE_DATE_EPOCH env var in this case? https://reproducible-builds.org/docs/source-date-epoch/
Maybe regrtest can detect SOURCE_DATE_EPOCH and omit --randomize in this case?
There was a problem hiding this comment.
If SOURCE_DATE_EPOCH env var is set, open an issue, and I will update regrtest for that ;-)
There was a problem hiding this comment.
yep! looks like launchpad sets that at the very least -- I assume the debian build system would too
There was a problem hiding this comment.
yep! looks like launchpad sets that at the very least -- I assume the debian build system would too
Yes, it's set by dpkg-buildpackage in all package builds.
|
This commit changed how some tests without the internet behave (in our build infrastructure, tests don't have access to the internet). For example, test that does need it is but after this change (bisecting led me here), it ends much more ominously with: It's not a huge deal, but probably not what was intended? |
|
Ah, it's probably the newly introduced default 10 minute timeout. No other codecmaps test fails like this because they all finish (are skipped) before 10 minutes, and when I add --timeout=1200, it works again. Sorry for the noise. |
|
You can add |
|
We are using It seems that codesmaps tests are not checking whether network resource is available? |
I checked. |
I think |
Oh ok, that makes sense. |
I know, that is what I wanted and why I was surprised that
Indeed, Thanks! With |
Add --fast-ci and --slow-ci options to libregrtest:
(skip slowest tests).
all tests).
regrtest header now lists test resources.
Makefile changes:
use --fast-ci option and TESTTIMEOUT variable.
became redundant with "--slow-ci".
and TESTTIMEOUT variable.
"find ... -print|xargs rm ...", same as "make clean".
GitHub Actions workflow:
options which became redundant with "--fast-ci".
Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
Tools/scripts/run_tests.py now uses --fast-ci option.
Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
📚 Documentation preview 📚: https://cpython-previews--109570.org.readthedocs.build/