Skip to content

Conversation

@RoniKishner
Copy link

Description

This PR fixes the double-counting of subtest failures in pytest's final test summary. Previously, when a test with subtests had failures, both the subtest failure and the top-level test failure were counted as regular "failed" tests, leading to inflated failure counts.

Changes

  • Modified pytest_report_teststatus hook in src/_pytest/subtests.py to return "subtests failed" category instead of "failed" for subtest failures
  • Updated _determine_main_color in src/_pytest/terminal.py to consider "subtests failed" when determining the main color (red)
  • Enhanced parse_summary_nouns in src/_pytest/pytester.py:
    • Updated regex to handle multi-word outcomes like "subtests failed"
    • Added pluralization mapping for subtest outcomes
    • Added whitespace handling for parsed outcomes
  • Updated test expectations in testing/test_subtests.py to match the new summary format

Example

Before:
===================== 2 failed, 2 subtests passed in 0.02s =====================
After:
===================== 1 failed, 2 subtests passed, 1 subtests failed in 0.02s =====================

This provides clear visibility into:

  • How many top-level tests failed (1 failed)
  • How many subtests failed (1 subtests failed)
  • How many subtests passed (2 subtests passed)

Closes #13986

@psf-chronographer psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Nov 20, 2025
Signed-off-by: Roni Kishner<rkishner@redhat.com>
@RoniKishner RoniKishner force-pushed the subtests-failure-info branch from 80ecce2 to 7fa74de Compare November 20, 2025 19:12
Copy link
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @RoniKishner for the PR.

I'm OK with separating subtest failures from normal failures.

I left a suggestion regarding the changelog, could you also take a look at the failing tests?

@@ -0,0 +1 @@
Fixed double-counting of subtest failures in the final test summary. Subtest failures are now reported separately as "subtests failed" instead of being counted as regular "failed" tests, providing clearer statistics. For example, a test with 3 subtests where 1 fails and 2 pass now shows "1 failed, 1 subtests failed, 2 subtests passed" instead of "2 failed, 2 subtests passed".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Fixed double-counting of subtest failures in the final test summary. Subtest failures are now reported separately as "subtests failed" instead of being counted as regular "failed" tests, providing clearer statistics. For example, a test with 3 subtests where 1 fails and 2 pass now shows "1 failed, 1 subtests failed, 2 subtests passed" instead of "2 failed, 2 subtests passed".
Show subtests failures separate from normal test failures in the final test summary.
Subtest failures are now reported separately as "subtests failed" instead of being counted as regular "failed" tests, providing clearer statistics.
For example, a test with 3 subtests where 1 fails and 2 pass now shows ``1 failed, 2 subtests passed, 1 subtests failed`` instead of ``2 failed, 2 subtests passed``.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix double-counting of subtest failures in test summary

2 participants