Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance assertion output for assert_all_called #224

Merged
merged 1 commit into from
Nov 29, 2022

Conversation

sileht
Copy link
Contributor

@sileht sileht commented Nov 28, 2022

When AllCalledAssertionError is raised its hard to guess which route got
called and which routes didn't get called.

This change aims to provide more details by listing not called routes
and by leveraging the repr of route object.

@codecov-commenter
Copy link

codecov-commenter commented Nov 28, 2022

Codecov Report

Base: 100.00% // Head: 100.00% // No change to project coverage 👍

Coverage data is based on head (b9df437) compared to base (6f8d05f).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #224   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           20        20           
  Lines         2757      2755    -2     
  Branches       417       417           
=========================================
- Hits          2757      2755    -2     
Impacted Files Coverage Δ
respx/models.py 100.00% <ø> (ø)
respx/router.py 100.00% <100.00%> (ø)
tests/test_mock.py 100.00% <100.00%> (ø)
tests/test_transports.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Owner

@lundberg lundberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a most welcome enhancement @sileht !

respx/router.py Outdated
raise AllCalledAssertionError("RESPX: some routes were not called!")
not_called_routes = {str(route) for route in self.routes if not route.called}
if not_called_routes:
formatted_not_called_routes = "* " + "\n* ".join(not_called_routes)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can make use of regular failed assertion output, instead of manual rendering 🤔

I'm thinking that this would give us the desired output:

assert not_called_routes == set(), "RESPX: some routes were not called!"

@sileht
Copy link
Contributor Author

sileht commented Nov 28, 2022

Yeah, this looks even nicer. Here is a pytest example:

    ...
    def assert_all_called(self) -> None:
        not_called_routes = [route for route in self.routes if not route.called]
>       assert not_called_routes == [], "RESPX: some routes were not called!"
E       AssertionError: RESPX: some routes were not called!
E       assert [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>] == []
E         Left contains 6 more items, first extra item: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>
E         Full diff:
E           [
E         -  ,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>,
E           ]

I have to use a list instead of a set as the Route object is not hashable.

@sileht
Copy link
Contributor Author

sileht commented Nov 28, 2022

Another example, if we also list called routes with:

       called_routes = [route for route in self.routes if route.called]
       assert called_routes == list(self.routes), "RESPX: some routes were not called!"
    def assert_all_called(self) -> None:
        called_routes = [route for route in self.routes if route.called]
>       assert called_routes == list(self.routes), "RESPX: some routes were not called!"
E       AssertionError: RESPX: some routes were not called!
E       assert [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>] == [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>]
E         At index 3 diff: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>> != <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>
E         Right contains 6 more items, first extra item: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>
E         Full diff:
E           [
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>,
E           ]

.tox/py310/lib/python3.10/site-packages/respx/router.py:106: AssertionError

Tell me which one you prefer and I will update the pull request.

@lundberg
Copy link
Owner

I have to use a list instead of a set as the Route object is not hashable.

👍

Tell me which one you prefer and I will update the pull request.

I'd say the first implementation aligns better with the assertion, i.e. all called == no remaining non-called

A third alternative that might give roughly the same output would be ...

assert len(non_called_routes) == 0

@sileht
Copy link
Contributor Author

sileht commented Nov 29, 2022

So it's ready 😀

@lundberg
Copy link
Owner

So it's ready 😀

Almost .. We should probably drop the AllCalledAssertionError class and adjust tests using that one.

@lundberg
Copy link
Owner

Once AllCalledAssertionError is dropped, all tests that are using it should be constrained to match part of the assertion message.

When AllCalledAssertionError is raised its hard to guess which route got
called and which routes didn't get called.

This change aims to provide more details by listing not called routes
and by leveraging the __repr__ of route object.
@lundberg lundberg changed the title chore: add more informations for AllCalledAssertionError Enhance assertion output for assert_all_called Nov 29, 2022
@lundberg lundberg merged commit 59e629e into lundberg:master Nov 29, 2022
@lundberg
Copy link
Owner

Thanks @sileht

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants