@autoretry generates multiple scenarios in JSON result

[lawnmowerlatte]

When using the new @autoretry functionality in combination with the JSON formatter, the JSON output contains multiple tests, one for each run regardless of the final result. This is different from the JUnit output which only contains the final run (pass or fail). You can see a demonstration of this in the sample code here. Simply run tox:

0 features passed, 1 failed, 0 skipped
1 scenario passed, 1 failed, 0 skipped
1 step passed, 1 failed, 0 skipped, 0 undefined
Took 0m0.000s

You can see the XML output (reports/TESTS-test.xml) starts with:

<testsuite errors="0" failures="1" name="test.Show autoretry creates multiple JSON entries" skipped="0" tests="2" time="0.000323">

Whereas the JSON output (reports/report_behave.json) contains 5 items under elements:

... "name": "I have a test that fails three times" ...
... "name": "I have a test that fails three times" ...
... "name": "I have a test that fails three times" ...
... "name": "I have a test that fails twice before passing" ...
... "name": "I have a test that fails twice before passing" ...

Is this the expected behavior? Is there some way to configure the JSON formatter? I suspect this is because the JSON output is a formatter (as opposed to the JUnit report).

[jenisys]

The @autoretry mechanism is a convenience hack. The JSON formatter shows you what gets really executed while the JUnit XML output only contains the compressed summary which scenarios are run and with which result.