Unify ways to run the Python test suite #109566

vstinner · 2023-09-19T10:28:13Z

There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO python -m test should be enough to fit all use cases.

Maybe we need a --ci, --buildbot, or --strict option which would run tests is "strict" mode.

Portable way to run tests:

./python -m test
./python -m test.regrtest
./python -m test.libregrtest
./python -m test.autotest

./python Lib/test/regrtest.py
./python Lib/test/autotest.py

./python -c 'from test import autotest'

The main drawback is that running ./python doesn't work when Python is built with ./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ./python.exe must be run instead. On Windows, it's just python.

Unix, from Python source code:

make buildbottest TESTOPTS="..."
make test
make testall
make testuniversal
make hostrunnertest

./python Tools/scripts/run_tests.py

Windows, from Python source code:

PCbuild\rt.bat
Tools\buildbot\test.bat

Lib/test/autotest.py just runs test.libregrtest.main().
Lib/test/regrtest.py runs test.libregrtest.main() and remove Lib/test/ from sys.path and make __file__ an absolute path.
PCbuild\rt.bat:
- python: pass -u -Wd -E -bb options
- regrtest: pass options passed to rt.bat (except of options specific to rt.bat)
- most options are to get the path to the python.exe program in PCbuild/
Tools\buildbot\test.bat runs PCbuild\rt.bat:
- rt.bat: pass -q -d options.
- regrtest: pass -j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
- ARM32 SSH pass -unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.
make buildbottest runs Tools/scripts/run_tests.py:
- python: pass $(TESTPYTHONOPTS) options.
- regrtest: pass -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.
Tools/scripts/run_tests.py:
- python: pass -u -W default -bb -E options -- also cross-compilation options, not detailed here
- regrtest: pass -r -w -j0 -u all,-largefile,-audio,-gui options, add -n on Windows.

GitHub Action workflow uses, .github/workflows/build.yml:

Windows x86: .\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
Windows x64: .\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
macOS: make buildbottest TESTOPTS="-j4 -uall,-cpu"
Ubuntu: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
Address Sanitizer: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"

Buildbot use, master/custom/factories.py:

UnixBuild and UnixCrossBuild: make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
UnixInstalledBuild: /path/to/python -m test ... with code to select -m test options.
BaseWindowsBuild: Tools\buildbot\test.bat ... with code to select .... options

Linked PRs

The text was updated successfully, but these errors were encountered:

* Add "make fastcitest" * --fast-ci has a default timeout to 10 minutes, --slow-ci has a default ot 20 minutes. * "make buildbottest" now uses "--slow-ci". Remove now redundant options. * GitHub Actions workflow now uses "make fastcitest". Remove now redundant options. Just use -j0: detect the number of CPUs.

* --fast-ci has a default timeout to 10 minutes, --slow-ci has a default ot 20 minutes. * Add "make fastcitest": similar to "make buildbottest", but use --fast-ci option instead of --slow-ci, and don't use TESTTIMEOUT. * "make buildbottest" now uses "--slow-ci". Remove now redundant options. * GitHub Actions workflow now uses "make fastcitest". Remove now redundant options. Just use -j0: detect the number of CPUs.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * Add "make fastcitest" target: similar to "make buildbottest", but use --fast-ci option instead of --slow-ci, and don't use TESTTIMEOUT. * "make buildbottest" now uses "--slow-ci". Remove now redundant options. * GitHub Actions workflow now uses "make fastcitest". Remove now redundant options. Just use -j0: detect the number of CPUs.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * Add "make fastcitest" target: similar to "make buildbottest", but use --fast-ci option instead of --slow-ci, and don't use TESTTIMEOUT. * "make buildbottest" now uses "--slow-ci". Remove now redundant options. * GitHub Actions workflow now uses "make fastcitest". Remove now redundant options. Just use -j0: detect the number of CPUs. * Set TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test" now uses --fast-ci option and TESTTIMEOUT variable. * "make hostrunnertest" now uses --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make coverage-report" now uses --fast-ci option and TESTTIMEOUT variable. * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow now uses "make test". Remove options which became redundant with "--fast-ci". Just use -j0: detect the number of CPUs. * Set TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test" now uses --fast-ci option and TESTTIMEOUT variable. * "make hostrunnertest" now uses --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make coverage-report" now uses --fast-ci option and TESTTIMEOUT variable. * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

regrtest now replaces the process with a new process to add options to the Python command line. Add options: "-u -W default -bb". When --fast-ci and --slow-ci options are used, the -E option is also added. The following methods to run the Python test suite don't replace the process: * "import test.autotest" * "from test.regrtest import main; main()" Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.

regrtest now replaces the process with a new process to add options to the Python command line. Add options: "-u -W default -bb". When --fast-ci or --slow-ci option is used, the "-E" option is also added. The following methods to run the Python test suite don't replace the process: * "import test.autotest" * "from test.regrtest import main; main()" Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.

When --fast-ci or --slow-ci option is used, regrtest now replaces the current process with a new process to add "-u -W default -bb -E" options to Python. The following methods to run the Python test suite don't replace the process: * "import test.autotest" * "from test.regrtest import main; main()" * "from test.libregrtest.main import main; main()" Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.

When --fast-ci or --slow-ci option is used, regrtest now replaces the current process with a new process to add "-u -W default -bb -E" options to Python. Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.

* On Windows, use subprocess.run() instead of os.execv(). * Only add needed options * Rename reexec parameter to _add_python_opts. * Rename --no-reexec option to --dont-add-python-opts.

WASM and WASI buildbots use multiple PYTHON environment variables such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the --python=COMMAND option is used.

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

…#109567)

…n#109570) * Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

When --fast-ci or --slow-ci option is used, regrtest now replaces the current process with a new process to add "-u -W default -bb -E" options to Python. Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.

* On Windows, use subprocess.run() instead of os.execv(). * Only add needed options * Rename reexec parameter to _add_python_opts. * Rename --no-reexec option to --dont-add-python-opts.

…9954) WASM and WASI buildbots use multiple PYTHON environment variables such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the --python=COMMAND option is used.

…ython#109969) regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

In the subprocess code path, wait until the child process completes with a timeout of EXIT_TIMEOUT seconds. Fix create_worker_process() regression: use start_new_session=True if USE_PROCESS_GROUP is true. WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of 30 seconds.

…0062) In the subprocess code path, wait until the child process completes with a timeout of EXIT_TIMEOUT seconds. Fix create_worker_process() regression: use start_new_session=True if USE_PROCESS_GROUP is true. WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of 30 seconds.

vstinner · 2023-09-29T02:06:21Z

Python options and regrtest options are now handled by new --fast-ci and --slow-ci options of regrtest.

Replace "--fail-env-changed --fail-rerun" with "--fast-ci". Tools/buildbot/test.bat pass --slow-ci which has the priority over --fast-ci.

The --nowindow option is deprecated and does nothing but logs a warning.

Remove "make testall" target: use "make buildbottest" instead.

Replace "--fail-env-changed --fail-rerun" with "--fast-ci". Tools/buildbot/test.bat pass --slow-ci which has the priority over --fast-ci.

Remove "make testall" target: use "make buildbottest" instead.

The --nowindows option is deprecated and does nothing but logs a warning.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023

pythongh-109566: Run buildbot tests with --fail-rerun

8dd5a56

bedevere-app bot mentioned this issue Sep 19, 2023

gh-109566: Run buildbot tests with --fail-rerun #109567

Merged

bedevere-app bot mentioned this issue Sep 19, 2023

gh-109566, regrtest: Add --fast-ci and --slow-ci options #109570

Merged

vstinner added a commit that referenced this issue Sep 19, 2023

gh-109566: Run GHA and buildbot tests with --fail-rerun (#109567)

67d9363

vstinner added the tests Tests in the Lib/test dir label Sep 20, 2023

vstinner added a commit to vstinner/cpython that referenced this issue Sep 22, 2023

pythongh-109566: Fix typo in PCbuild/rt.bat

5d1c46f

bedevere-app bot mentioned this issue Sep 22, 2023

gh-109566: Fix typo in PCbuild/rt.bat #109701

Merged

vstinner added a commit that referenced this issue Sep 22, 2023

gh-109566: Fix typo in PCbuild/rt.bat (#109701)

3cce6be

bedevere-app bot mentioned this issue Sep 26, 2023

gh-109566: regrtest reexecutes the process #109909

Merged

bedevere-app bot mentioned this issue Sep 26, 2023

gh-109566: Fix regrtest code adding Python options #109926

Merged

bedevere-app bot mentioned this issue Sep 27, 2023

gh-109566: Fix regrtest Python options for WASM/WASI #109954

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Sep 27, 2023

pythongh-109566: regrtest doesn't enable --rerun if --python is used

13ccc6d

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

bedevere-app bot mentioned this issue Sep 27, 2023

gh-109566: regrtest doesn't enable --rerun if --python is used #109969

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Sep 27, 2023

pythongh-109566: regrtest doesn't enable --rerun if --python is used

b61593e

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

vstinner added a commit that referenced this issue Sep 27, 2023

gh-109566: regrtest doesn't enable --rerun if --python is used (#109969)

62881a7

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023

pythongh-109566: Run GHA and buildbot tests with --fail-rerun (python…

9e4d048

…#109567)

csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023

pythongh-109566: Fix typo in PCbuild/rt.bat (python#109701)

4ba8474

csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023

pythongh-109566: regrtest doesn't enable --rerun if --python is used (p…

5e4b590

…ython#109969) regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.

bedevere-app bot mentioned this issue Sep 29, 2023

gh-109566: regrtest _add_python_opts() handles KeyboardInterrupt #110062

Merged

vstinner closed this as completed Sep 29, 2023

vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023

pythongh-109566: PCbuild/rt.bat now uses --fast-ci

292fa12

Replace "--fail-env-changed --fail-rerun" with "--fast-ci". Tools/buildbot/test.bat pass --slow-ci which has the priority over --fast-ci.

bedevere-app bot mentioned this issue Sep 29, 2023

gh-109566: PCbuild/rt.bat now uses --fast-ci #110120

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023

pythongh-109566: regrtest --fast-ci no longer enables --nowindow

6efcff1

The --nowindow option is deprecated and does nothing but logs a warning.

bedevere-app bot mentioned this issue Sep 29, 2023

gh-109566: regrtest --fast-ci no longer enables --nowindow #110121

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023

pythongh-109566: Remove make testall

8725c01

Remove "make testall" target: use "make buildbottest" instead.

bedevere-app bot mentioned this issue Sep 29, 2023

gh-109566: Remove make testall #110122

Merged

vstinner added a commit that referenced this issue Sep 29, 2023

gh-109566: PCbuild/rt.bat now uses --fast-ci (#110120)

f3df8fa

Replace "--fail-env-changed --fail-rerun" with "--fast-ci". Tools/buildbot/test.bat pass --slow-ci which has the priority over --fast-ci.

vstinner added a commit that referenced this issue Sep 29, 2023

gh-109566: Remove make testall (#110122)

9c91141

Remove "make testall" target: use "make buildbottest" instead.

vstinner added a commit that referenced this issue Sep 29, 2023

gh-109566: regrtest --fast-ci no longer enables --nowindows (#110121)

5ae6c6d

The --nowindows option is deprecated and does nothing but logs a warning.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify ways to run the Python test suite #109566

Unify ways to run the Python test suite #109566

vstinner commented Sep 19, 2023 •

edited by bedevere-app bot

vstinner commented Sep 29, 2023

Unify ways to run the Python test suite #109566

Unify ways to run the Python test suite #109566

Comments

vstinner commented Sep 19, 2023 • edited by bedevere-app bot

Linked PRs

vstinner commented Sep 29, 2023

vstinner commented Sep 19, 2023 •

edited by bedevere-app bot