Add an option to the test runner to exit immediately upon the first failure

[embray]

Inspired by the --exitfirst option to py.test, this adds a --exitfirst option to Sage's test runner to exit immediately upon the first test failure.

I've wanted something like this at several points, but one reason in particular is that part of the build of the patchbot Docker image is to perform the baseline test run, so that the image itself can be reused without having to rebuild the baseline each time. Obviously, if a test fails during the baseline test run then I want the Docker image build to fail immediately, and not wait until the rest of the test suite has run.

Component: doctest framework

Author: Erik Bray

Branch/Commit: c6a5b40

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/22832

[embray]

New commits:

c12a4d3	Initial attempt to add a test runner option to stop a test run as soon as a single failure is encountered.
2f359e8	Make --fail-once work with parallel doctests.

[embray]

Commit: 2f359e8

[embray]

Author: Erik Bray

[embray]

Branch: u/embray/tests/fail-once

[jdemeyer]

Replying to @embray:

Inspired by the --exitfirst option to py.test, this adds a --fail-once option to Sage's test runner to exit immediately upon the first test failure.

Obvious question: why not call the Sage option exitfirst? I find it a better name than fail-once.

[jdemeyer]

comment:3

I'm making some changes, hang on...

[jdemeyer]

Changed branch from u/embray/tests/fail-once to u/jdemeyer/tests/fail-once

[jdemeyer]

Changed commit from 2f359e8 to c2bff84

[jdemeyer]

New commits:

f26f64e	Initial attempt to add a test runner option to stop a test run as soon as a single failure is encountered.
2e9a80c	Make --fail-once work with parallel doctests.
c2bff84	Rename --fail-once to --exitfirst; minor fixes; add test

[jdemeyer]

Reviewer: Jeroen Demeyer

[embray]

comment:6

I'm fine with that, thanks.

[embray]

comment:7

Replying to @jdemeyer:

Replying to @embray:

Inspired by the --exitfirst option to py.test, this adds a --fail-once option to Sage's test runner to exit immediately upon the first test failure.

Obvious question: why not call the Sage option exitfirst? I find it a better name than fail-once.

I personally find "fail once" to be clearer than "exit first"--"Exit first what?" is what it makes me ask. That said, the documentation makes it clear enough, and I don't mind the consistency.

[embray]

comment:8

Oops, one little nitpick--you forgot to remove the line fail_immediately=False after renaming it to abort_now everywhere else.

[sagetrac-git]

Changed commit from c2bff84 to c6a5b40

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

c6a5b40

Remove forgotten line

[vbraun]

Changed branch from u/jdemeyer/tests/fail-once to c6a5b40