Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand CI pytest reporting config to ignore xfails #27807

Merged
merged 1 commit into from
Feb 21, 2024

Conversation

ksunden
Copy link
Member

@ksunden ksunden commented Feb 20, 2024

PR summary

Pytest 8 brought with it an expansion of the reporting for xfails (and xpasses)
which with our current config (-ra) on CI renders the reports rather difficult to
read. This is because we have quite a lot of xfails, for various reasons:

  • We have many that are very intentional and are testing our test fixures and explicitly pass strict=True
  • We have things like the recent templated out tests for units behavior that introduced many xfails at once

As a result seeing the output on xfails is not particularly useful for us, especially.

As x is included in the a (all) option, we are left to expand out all of the components that we do want, which is a little messy.

Unfortuntately there is not a toggle that still gets us the one line "this test xfails" without
providing the full output, which I probably would use if available.

I was borderline on whether or not to remove the xpasses (X). As there are not that many of them in my experience and output was not present at least for those that xpassed for me locally I decided that keeping the one line summary was useful.
My reading of the discussion around the addition of this feature to pytest indicates that output would be included if present, but since generally xpassing mostly means we are not getting an exception, I think cases where output is produced are likely relatively rare (and honestly most likely on the strict mode tests that would fail the suite and cause us to look in the first place).

I considered moving all of these to --add-opts in the pyproject.toml config, but opted to just do the minimal change that would not affect dev machines, just CI.

PR checklist

@github-actions github-actions bot added the CI: Run cygwin Run cygwin tests on a PR label Feb 20, 2024
@QuLogic QuLogic added this to the v3.9.0 milestone Feb 21, 2024
@QuLogic QuLogic merged commit 88d64e5 into matplotlib:main Feb 21, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: Run cygwin Run cygwin tests on a PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants