Flag to run tests based on the summary output of a previous run

[nicoddemus]

What's the problem this feature will solve?

A common workflow is to re-run locally tests that have failed on CI. When there are a number of failures, it is not straightforward to do so, as it requires a lot of copy/paste test ids.

Describe the solution you'd like

If the user is running with -ra on CI, the summary output could be used as input to pytest to only run the failed tests. Then the user can copy/paste the summary output into a file, and call pytest with -x/--sw in order to resolve the tests bit by bit locally.

I think this could be done via a flag:

• --from-summary-output=file.txt
• --from-summary=file.txt

We don't need a short version for this flag I believe.

Additional context

Idea came from this discussion in the mailing list.

[RonnyPfannschmidt]

We should instead have options to fetch last failed from a artifact

[oscarbenjamin]

There are also uses for running a list of tests from a file or preserving --lf that don't involve CI.

It would be good to have a parallel option for saving the -ra output to the file that would be read by --from-summary

[nicoddemus]

We should instead have options to fetch last failed from a artifact

Not sure one things excludes the other TBH.

It would be good to have a parallel option for saving the -ra output to the file that would be read by --from-summary

Agreed.

[RonnyPfannschmidt]

i primarily want to avoid using a summary that intentionally doesnt have a defined stable output as data source - a reportlog, lastfailed file or junitxml report would be much better for consistent parsing

[nicoddemus]

I see.

One of the requirements is "copy/paste from the summary output" so users do not have to setup anything extra on CI. We would have to forego that requirement for that to work, in which case I would prefer to withdraw this proposal as it is no longer viable for the use case I have in mind (someone else of course can open a new proposal similar to this one).

[oscarbenjamin]

There just needs to be an option to output whatever format this is on stdout. The -ra option is not default anyway.

Needing to fiddle with artifacts is awkward though and should be avoided.