Show total progress in terminal during testing

[nicoddemus]

This is a proposal to change the terminal output while running tests to include the total tests ran so far and the total tests still to run, or percentage of the total tests executed so far.

Currently here's the normal pytest output (minus header):

============================= test session starts =============================
collected 1909 items

testing\acceptance_test.py ...............s................................x.............
testing\deprecated_test.py ......
testing\test_argcomplete.py ss
testing\test_assertion.py ........................................................................s.......
<snip>

What I propose is:

============================= test session starts ===============================
collected 1909 items

testing\acceptance_test.py ...............s............................ [44/1909]
                           ....x.............                           [62/1909]
testing\deprecated_test.py ......                                       [68/1909]
testing\test_argcomplete.py ss                                          [70/1909]
testing\test_assertion.py ............................................ [114/1909]
                          ............................s.......         [150/1909] 
<snip>

This would be a better view of the overall progress and how much tests are still left on the run.

Alternatively instead of showing test counts we could just show a percentage of the total run so far:

============================= test session starts ===============================
collected 1909 items

testing\acceptance_test.py ...............s................................ [ 2%]
                           ....x.........                                   [ 3%]
testing\deprecated_test.py ......                                           [ 3%]
testing\test_argcomplete.py ss                                              [ 3%]
testing\test_assertion.py ................................................. [ 5%]
                          ............................s.                    [ 7%] 
<snip>

Thoughts? cc @RonnyPfannschmidt @hpk42 @The-Compiler @hackebrot

Side note

This idea came to be because it Jenkins (it seems) only updates its console view when the process outputs a new line. When running with xdist, pytest only prints a . per test, only printing a new line at the end of the session, so the output windows doesn't update until the end of the run.

Changing the output to the one proposed here would fix that and also provide a nice user experience (IMO).

[nicoddemus]

Note: this would work with xdist outside of the box as well. 😉

[RonnyPfannschmidt]

doesnt pytest-sugar already implement a much nicer version of this

[nicoddemus]

Indeed, but pytest-sugar only works on Linux. 😬

[The-Compiler]

I like it! pytest-sugar also has a lot of other problems (it being very fancy is also causing issues), and good defaults matter.

[RonnyPfannschmidt]

according to Teemu/pytest-sugar#49 (comment) it only a codepage issue and solvable by a single command

[RonnyPfannschmidt]

well, im fine with enhancing terminal output, but i do wonder how much harder it will make the removal of the current terminalwriter mess

also note that currently the terminal width determination of pylib is completely broken for output capture situation

[nicoddemus]

according to Teemu/pytest-sugar#49 (comment) it only a codepage issue and solvable by a single command

Didn't know that, thanks for the link!

well, im fine with enhancing terminal output, but i do wonder how much harder it will make the removal of the current terminalwriter mess

AFAIK it won't make things worse (it will depend on tw.write as does all the other code which depends on terminalwriter) but I'm of course willing to abort the idea if the implementation gets too messy.

[ApaDoctor]

does anybody works on this issue?
i want to take it

[RonnyPfannschmidt]

@nicoddemus do you have any plans/attempts already started, if thats not the case, i propose we let @ApaDoctor try

[nicoddemus]

I have a work in progress at home actually, I don't think it is pushed to me fork though. @ApaDoctor let me push that branch when I get home so you can take a look. This is a little tricky than thought at first because we have to "hook" into terminal's writer to ensure we are tracking the correct number of chars written to the current line so far.

[nicoddemus]

@ApaDoctor take a look at https://github.com/nicoddemus/pytest/tree/console-progress-2657 if you are interested.

[Teemu]

This is pretty cool 👍

I think if this existed 4 years ago I would have never started pytest-sugar project.

[nicoddemus]

Which would be a shame because pytest-sugar terminal is awesome 😁

[kevinpohlmeier]

When I added a pytest.ini file with the following contents, I started seeing progress on my xdist runs:

[pytest]
log_cli = 1
log_cli_level = WARN