Invalid percentages of Test Suite progress

[merzlya]

Progress of Test Suite shows more than 100% if Teardown fails:

$ python3 -m pytest -v ./test.py &> res.txt; cat res.txt                                                                                                                                        
============================= test session starts ==============================                                                                                                                                   
platform linux -- Python 3.6.4, pytest-3.3.1, py-1.5.0, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .cache
rootdir: /home/pavel, inifile:
plugins: allure-adaptor-1.7.8
collecting ... collected 1 item

test.py::test_foo1 PASSED                                                [100%]
test.py::test_foo1 ERROR                                                 [200%]

==================================== ERRORS ====================================

test.py:

import pytest                                                                                                                                                                                                      

@pytest.fixture
def fail_teardown():
    print("SETUP")
    yield fail_teardown
    print("TEARDOWN")
    assert 1 == 2

def test_foo1(fail_teardown):
    print("TEST")

System:

$ pip list 2>/dev/null                                                                                                                                                                          
appdirs (1.4.3)                                                                                                                                                                                                    
asn1crypto (0.24.0)                                                                                                                                                                                                
attrs (17.4.0)
bcrypt (3.1.4)
cffi (1.11.2)
cryptography (2.1.4)
idna (2.6)
isc (2.0)
json2html (1.1.1)
lxml (4.1.1)                                                                                                                                                                                                       
msgpack-python (0.4.8)                                                                                                                                                                                             
namedlist (1.7)                                                                                                                                                                                                    
netifaces (0.10.6)                                                                                                                                                                                                 
packaging (16.8)                                                                                                                                                                                                   
paramiko (2.4.0)                                                                                                                                                                                                   
pip (9.0.1)                                                                                                                                                                                                        
pluggy (0.6.0)                                                                                                                                                                                                     
ply (3.10)                                                                                                                                                                                                         
py (1.5.0)                                                                                                                                                                                                         
pyasn1 (0.4.2)                                                                                                                                                                                                     
pycparser (2.18)                                                                                                                                                                                                   
PyNaCl (1.2.0)                                                                                                                                                                                                     
pyparsing (2.2.0)                                                                                                                                                                                                  
pyserial (3.4)                                                                                                                                                                                                     
pytest (3.3.1)                                                                                                                                                                                                     
pytest-allure-adaptor (1.7.8)                                                                                                                                                                                      
scp (0.10.2)                                                                                                                                                                                                       
setuptools (38.2.5)                                                                                                                                                                                                
six (1.11.0)                                                                                                                                                                                                       
speedtest-cli (1.0.7)                                                                                                                                                                                              
ws4py (0.3.4)                                                                                                                                                                                            
$ uname -a                                                                                                                                                                                      
Linux afitester 4.14.10-1-ARCH #1 SMP PREEMPT Fri Dec 29 20:17:35 UTC 2017 x86_64 GNU/Linux

[nicoddemus]

@merzlya thanks for the report!

[asottile]

it looks even stranger without -v:

$ py.test test.py
============================= test session starts ==============================
platform darwin -- Python 3.6.3, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /private/tmp/wat, inifile:
collected 1 item                                                               

test.py .                                                                [100%]E [200%]

[nicoddemus]

I realized that this is probably also giving wrong output for plugins like pytest-rerunfailures...

[nicoddemus]

This fix will need a new hook (#3102), but after that the solution is simple and I managed to fix it locally. 😁