add RaisesGroup & Matcher

[jakkdl]

Previous PR's: #11656 (ExpectedExceptionGroup), #11671 (closed to give this a fresh PR w/o stale comments)
fixes: #11538 #12504

This adds RaisesGroup (also exported as raises_group) and Matcher, to allow a robust way of expecting ExceptionGroup and the ability to specify the structure of nested groups, etc etc.

with RaisesGroups(ValueError):
    raise ExceptionGroup("", (ValueError(),))

This is the exact implementation available in trio.testing.RaisesGroup which has been in use since december 2023 and gone through a couple iterations since to settle on a good interface. The most recent addition was python-trio/trio#3145 which added error messages, where the formatting can probably still be fine-tuned.

Pytest currently has excinfo.group_contains for checking exceptiongroups... which is very problematic. I think it should be deprecated, either in the same release as this PR, or very soon after.

This currently does not touch the implementation of pytest.raises, but if we made Matcher a contextmanager we could easily support

with Matcher(ValueError):
    raise ValueError()

this would fix e.g. #12763 with Matcher having the check parameter. The only complication is the legacy form of pytest.raises but that should be solvable with a small helper function.
This can however be done in a followup PR.

This currently does not add an .assert_matches method, which previous iterations of the PR had. The reason to add that would be to improve the error message on failure, though it's possible to get the same result with

assert (m := Matcher(TypeError)).matches(e), m.fail_reason
# versus
Matcher(TypeError).assert_matches(e)

though note that .matches() is already a fairly niche feature that is entirely separate to the context-manager use (not to be confused with the match parameter).

TODO:

• bump the issue # on the newsfragment file
• documentation

[RonnyPfannschmidt]

We ought not to use such a generic name easily

There's a general wishlist plan to enable matchers as a whole that act similar to dirty equals

The matcher api that's being used here conflates multiple aspects of that in a manner that's problematic to partition later

Ideally the whole api around matchers is first evolved as a plugin to avoid early api lock in as it's hard to make good matchers

I'll elaborate further when the flu is over

[jakkdl]

sure, I can rename it... maybe something that's closer to RaisesGroup... RaisesExc/RaisesException? or something like ExpectedExc/ExpectedException

This has been available & in use for >1 year in trio, which has allowed the interface to mature. I don't think pytest needs to repeat that process by putting it in a plugin.

I know you've mentioned matcher objects in #11538 and I'm excited for what that might entail in the future, but I don't think we want to wait for that to add usable exceptiongroup support. If you have concrete ways of improving the RaisesGroup interface to harmonize with it I'm of course open to suggestions though.

[Zac-HD]

I've been heavily involved in the design and review for this feature as we've developed and shipped it in Trio. I'm excited to get it into Pytest (and ideally soonish), but I shouldn't be the only approving core dev for a substantial feature.

[jakkdl]

@Zac-HD please review the documentation :)

It feels a bit silly to add big warning boxes to ExceptionInfo.group_contains and not deprecate it, but it also feels silly to deprecate it one release after adding it.

[Zac-HD]

@Tusenka unfortunately pull requests are the wrong place to ask questions like this, unless you're either the author of the PR or a maintainer of the package. I suggest StackOverflow instead.

[jakkdl]

please bring basic questions like these to StackOverFlow, the python discord, ask an LLM of your choice, or otherwise. This is not relevant to the PR at hand and only wastes the time of maintainers without any benefit to pytest.

[jakkdl]

I think I'm mostly just waiting for a quick pass on docs changes from @Zac-HD (or others), and otherwise this can be merged and I can look at potential followup for pytest.raises and/or deprecating excinfo.group_contains

[jakkdl]

so, uh, sorry to everybody reviewing this, cause this last commit is a monster. I noticed a ton of small random things (though I probably shouldn't have bothered with backquotes tbh)

Anyway, this gets rid of RaisesContext and simplifies pytest.raises a lot. There were a couple niche things I hadn't incorporated into RaisesExc/RaisesGroup (tuples of exceptions, generics) and a bunch of differences in formatting.
One of the bigger changes is whether an unexpected exception is let through or not - when the matching is non-trivial I opted to change it so that RaisesExc/RaisesGroup always raised Failed, with the original exception as context, partly I think that's a superior interface, but also because I needed to output the reason why the match failed somewhere.

There's also a question of AssertionError vs Failed - currently RaisesExc will raise AssertionError on any failed match for the sake of minimizing the diff in tests, but I think Failed is better.

If any of this stuff is controversial I'll break stuff off into a separate PR, and I was hoping to do some of this in a follow-up PR, but it ended up becoming relevant to the discussion so I just did it in here for now.

[nicoddemus]

Thanks a lot @jakkdl for the great work here!

I'm OK with us having pytest.RaisesGroup and it being different than pytest.raises, you have made your points and since nobody else has given their opinions, I guess it is no big deal in the end.

However I think we should reconsider also exposing pytest.raises_group, as having both ways to do the exact same thing might not be a net positive? So I propose just going ahead and having only pytest.RaisesGroup, leaving out the pytest.raises_group alias.

[nicoddemus]

Hi @jakkdl, thanks for sticking with it so far!

Can you just fix the conflict please? Then I will be happy to merge. 👍

[jakkdl]

lel, amusing how big a conflict you can get from a simple be2600e

I'll merge before it happens again! 🎉 🚀 🎉 🚀

[jakkdl]

ugh I just noticed this didn't quite have 100% patch coverage. We should enforce that in config

[nicoddemus]

Thanks @jakkdl.

In the future, please consider squashing the commits before merging, as this PR introduced a lot of commits which don't make sense separately.

See https://github.com/pytest-dev/pytest/blob/main/CONTRIBUTING.rst#mergesquash-guidelines. 👍

[jakkdl]

oh shit, I'm way too used to squashing being the default

do you want me to squash them on master (if I can?), or will you do that?

[RonnyPfannschmidt]

It's already on main and main is protected

[nicoddemus]

Yeah, unfortunately now it has landed on main so we cannot change it, as a Git good practice.

Not the end of the world, but something to take care in the future. Thanks again!