Mocking fails when using Authlib HTTPX integration OAuth2 client

[bobh66]

I'm trying to mock the responses from Keycloak using the Authlib HTTPX integration client for OAuth2.

I'm getting the following exception because the auth_flow() hook for the OAuth2 client is rewriting the request.url attribute so it is no longer a URLResponse object, it's a plain URL (again).

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/authlib/integrations/httpx_client/oauth2_client.py:109: in _fetch_token
    auth=auth, **kwargs)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1316: in post
    timeout=timeout,
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/authlib/integrations/httpx_client/oauth2_client.py:89: in request
    method, url, auth=auth, **kwargs)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1097: in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/respx/mocks.py:162: in unbound_async_send
    return await self.__AsyncClient__send__spy(client, request, **kwargs)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/respx/mocks.py:533: in __AsyncClient__send__spy
    response = await _AsyncClient__send(client, request, **kwargs)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1118: in send
    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1148: in send_handling_redirects
    request, auth=auth, timeout=timeout, history=history
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1184: in send_handling_auth
    response = await self.send_single_request(request, timeout)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/client.py:1208: in send_single_request
    response = await dispatcher.send(request, timeout=timeout)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/dispatch/connection_pool.py:157: in send
    raise exc
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/dispatch/connection_pool.py:153: in send
    response = await connection.send(request, timeout=timeout)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/dispatch/connection.py:42: in send
    self.connection = await self.connect(timeout=timeout)
/home/bobh/.virtualenvs/traffica_stc/lib/python3.7/site-packages/httpx/dispatch/connection.py:63: in connect
    host, port, ssl_context, timeout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <respx.mocks.HTTPXMock object at 0x7f1e34a9a898>
hostname = 'myhost', port = 443
ssl_context = <ssl.SSLContext object at 0x7f1e33f9b8b8>
timeout = Timeout(timeout=5.0)

    async def __Backend__open_tcp_stream__mock(
        self,
        hostname: str,
        port: int,
        ssl_context: typing.Optional[ssl.SSLContext],
        timeout: Timeout,
    ) -> BaseSocketStream:
        response = getattr(hostname, "attachment", None)  # Pickup attached template
>       return await response.socket_stream
E       AttributeError: 'NoneType' object has no attribute 'socket_stream'

By this time the response is gone so I don't know what the mocked method could return. It would have to go back and re-match on the request (somehow).

Any ideas to work around or fix this? I'm happy to push a PR if someone can point me at a possible fix.

Thanks

[lundberg]

Interesting, and thanks for catching this @bobh66.

I haven't used the authlib integration client myself, but since it is using httpx Auth it should be possible, hopefully solving any other auth implementations aswell.

Quickly looked the authlib/httpx code around this, and the auth instance is passed along to Client.send, which we already have under control. I guess we could use mock.patch.object to spy on auth.auth_flow and re-build the URLResponse using the rebuilt yielded request and the original request's URLResponse.

Don't think we need authlib to test and implement this. Just need a custom TestAuth which could re-set the request.url to a regular URL to simulate this.

[lundberg]

We might need to change to patching client.send_handling_redirects instead of client.send, so we don't need to handle different AuthTypes.

[bobh66]

client.send_handling_redirects() calls client.send_handling_auth() which calls the auth_flow() method that clobbers the URL. client.send_single_request() might be the place to patch to ensure that all of the auth changes have happened and there is nothing left that will change the url response.

[StephenBrown2]

For future @lundberg this is something that may/will change if/when Middleware becomes a thing: encode/httpx#800

[bobh66]

I hacked something that works by mocking send_single_request but since not all clients use send_single_request it's not a good solution, the unit tests fail miserably with that code in.

[lundberg]

Awesome that you're adding this to the context @StephenBrown2 👍🏻

Just for background info, I've implemented the response patching as deep as possible to minimize httpx internal api changes, and also handle testing custom implementations. This is true for the async path, but unfortunately not for sync, since it is using urlib3.

[firesock]

Just to chime in quickly with a workaround based on #46 (comment)

from authlib.integrations.httpx_client.oauth2_client import OAuth2Auth, OAuth2ClientAuth

@pytest.fixture(name="sso")
def fixture_sso(monkeypatch):
    def auth_flow_spy(auth_class):
        original_auth_flow = auth_class.auth_flow

        @functools.wraps(original_auth_flow)
        def f(auth, request):
            respx_url = request.url
            authlib_request = next(original_auth_flow(auth, request))
            authlib_request.url = respx_url
            yield authlib_request

        return f

    monkeypatch.setattr(OAuth2Auth, "auth_flow", auth_flow_spy(OAuth2Auth))
    monkeypatch.setattr(OAuth2ClientAuth, "auth_flow", auth_flow_spy(OAuth2ClientAuth))

It solved our particular needs, and might be helpful to others until a proper fix is in place

[lundberg]

@bobh66, a new release is out where I patch httpcore instead of httpx internals and hopefully this solves your issue. Haven't tried, so I'm not sure, but please try it out your self and report back if it solves or not.

[lundberg]

@firesock, would be awesome if you also could try the latest release to see if it solves this issue.

[moubctez]

This seems to be solved.

[firesock]

Sorry, only got around to updating our pin recently. I can confirm with authlib 0.15.1, httpx 0.16.1 and respx 0.14.0 the workaround is unnecessary.