Expand CI pytest reporting config to ignore xfails #27807
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR summary
Pytest 8 brought with it an expansion of the reporting for xfails (and xpasses)
which with our current config (
-ra
) on CI renders the reports rather difficult toread. This is because we have quite a lot of xfails, for various reasons:
strict=True
As a result seeing the output on xfails is not particularly useful for us, especially.
As
x
is included in thea
(all) option, we are left to expand out all of the components that we do want, which is a little messy.Unfortuntately there is not a toggle that still gets us the one line "this test xfails" without
providing the full output, which I probably would use if available.
I was borderline on whether or not to remove the xpasses (
X
). As there are not that many of them in my experience and output was not present at least for those that xpassed for me locally I decided that keeping the one line summary was useful.My reading of the discussion around the addition of this feature to pytest indicates that output would be included if present, but since generally xpassing mostly means we are not getting an exception, I think cases where output is produced are likely relatively rare (and honestly most likely on the strict mode tests that would fail the suite and cause us to look in the first place).
I considered moving all of these to
--add-opts
in thepyproject.toml
config, but opted to just do the minimal change that would not affect dev machines, just CI.PR checklist