combination of --testmon and --tlf will execute failing tests multiple time in some circumstances

[TauPan]

I have a situation where executing pytest --testmon --tlf module1/module2 after deleting .testmondata will yield 3 failures.

The next run of pytest --testmon --tlf module1/module2 yields 6 failures (3 duplicates of the first 3) and each subsequent run yields 3 more duplicates.

If I run pytest --testmon --tlf module1/module2/tests I get back to 3 tests, the duplication starts as soon as I remove the final tests in the path (or execute with the root path of the repository).

I've tried to find a simple way to reproduce this but failed so far. I've also tried looking at what changes in .testmondata but didn't spot anything.

I'd be grateful if you could give me a hint how I might reproduce or analyze this, as I unfortunately can't share the code at the moment.

[TauPan]

This also happens with --last-failed occasionally, although it's not as easy to reproduce. This might be an instance of #92 or similar to it.

I happen to be able to reproduce this be resetting an unstaged change via git that deleted more lines than it added and affected the failing tests.

[blueyed]

Sounds like #102 might help here.

btw: you can usually differentiate between cached results and executed ones by looking at the color highlighting of the tracebacks - cached ones are not colored.

[tarpas]

@TauPan could you please try with 39ee38a ? There is an assert which could tell us more about the circuimstances.

[TauPan]

Uhm, as far as I can see the assert is never false. I've put a pdb breakpoint before it and the first time report.when == 'setup' and the second time it's == 'call'. I see the same test reported twice with -rapP (both times as FAIL).

(tried this with 39ee38a and master)

However I don't get multiplying test failures any more, just one cached failure OR pass (didn't notice that one before, passes are also cached).

=================================================================================== short test summary info ===================================================================================
FAIL project/package/tests/test_subject_models.py::TestSUBJECT2AScore::()::test_stuff[fixture_value]
FAIL project/package/tests/test_subject_models.py::TestSUBJECT2AScore::()::test_stuff[fixture_value]

I'm not sure what I'm looking at here. Do you need full the redacted log with -rapP or something?

[blueyed]

@TauPan

the assert is never false

I've seen this in #103 (comment).

[TauPan]

@TauPan

the assert is never false

I've seen this in #103 (comment).

FWIW I can't reproduce the assertion failure by hitting Ctrl-C after
the first failed test.

[TauPan]

I just hit a backtrace. I had an error in multiple assertions (3 tests, switched from asserting is None to is Decimal('...') which failed then corrected it to == Decimal('...').

There was no Ctrl-C at any point, i.e. I waited out all those test runs.

assertion-failure-redacted.txt

[tarpas]

It's not completely explained but v0.9.12 should be more robust in this regard, can you confirm?

[TauPan]

It's not completely explained but v0.9.12 should be more robust in this regard, can you confirm?

Yup, I can't reproduce this any more. Thanks!