Cannot inject HTTPX client as pytest fixture #185

gregbrowndev · 2022-08-18T14:16:40Z

Hi,

Thanks for fixing the previous issue super quickly. However, I've noticed a json.decoder.JSONDecodeError problem when you inject HTTPX' AsyncClient as a pytest fixture. If you put the @mocketize decorator on the fixture it fixes the problem. However, this seems a bit odd.

import asyncio
import json

import httpx
import pytest
from httpx import AsyncClient
from mocket import mocketize
from mocket.mockhttp import Entry


@pytest.fixture
def httpx_client() -> AsyncClient:
    # Note: should use 'async with'
    return httpx.AsyncClient()


async def send_request(client: AsyncClient, url: str) -> dict:
    r = await client.get(url)
    return r.json()


@mocketize
def test_mocket(
    httpx_client: AsyncClient
):
    # works if you define httpx_client locally:
    # httpx_client = httpx.AsyncClient()
    url = "https://example.org/"
    data = {"message": "Hello"}

    Entry.single_register(
        Entry.GET,
        url,
        body=json.dumps(data),
        headers={'content-type': 'application/json'}
    )

    loop = asyncio.get_event_loop()
    
    coroutine = send_request(httpx_client, url)
    actual = loop.run_until_complete(coroutine)
    
    assert data == actual

Stacktrack

/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/asyncio/base_events.py:646: in run_until_complete
    return future.result()
test_mocket.py:22: in send_request
    return r.json()
/Users/gregorybrown/Library/Caches/pypoetry/virtualenvs/tariff-management-6yv2RoDp-py3.10/lib/python3.10/site-packages/httpx/_models.py:743: in json
    return jsonlib.loads(self.text, **kwargs)
/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x10c48b8e0>
s = '<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset="utf-8" />\n    <meta http-eq...on.</p>\n    <p><a href="https://www.iana.org/domains/example">More information...</a></p>\n</div>\n</body>\n</html>\n'
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/json/decoder.py:355: JSONDecodeError

Also, I know there's a lot of weird/questionable stuff going on in this test but I'm just trying to verify the behaviour around using async HTTPX within a sync app.

Thanks

The text was updated successfully, but these errors were encountered:

mindflayer · 2022-08-18T14:22:23Z

From your stacktrace I don't understand how mocket would be involved with this error. It seems like something happening on httpx side.

Could you please comment all mocket's code?

mindflayer · 2022-08-18T14:24:15Z

Also, I see you are not using async_mocketize but your code looks like async to me.

gregbrowndev · 2022-08-18T14:31:51Z

Thanks for the quick reply.

Maybe as a bit of context. This test is trying to emulate a sync function that uses httpx to make a bunch of concurrent requests. Basically calling async from a sync function. It does this by putting the coroutines on the event loop like in the test. The sync function is buried deep in the app, so this is to contain the async and prevent us converting the whole app to async in one go.

The pytest is synchronous, hence it doesn't require async_mocketize. For example, this test passes simply because httpx_client is instantiated locally:

@mocketize
def test_mocket():
    httpx_client = httpx.AsyncClient()
    url = "https://example.org/"
    data = {"message": "Hello"}

    Entry.single_register(
        Entry.GET,
        url,
        body=json.dumps(data),
        headers={'content-type': 'application/json'}
    )

    loop = asyncio.get_event_loop()
    
    coroutine = send_request(httpx_client, url)
    actual = loop.run_until_complete(coroutine)
    
    assert data == actual

The registered mocks using Entry.single_register are being returned by the httpx requests as expected with this set up. The only problem is the test breaks if you use a fixture.

gregbrowndev · 2022-08-18T14:34:10Z

From your stacktrace I don't understand how mocket would be involved with this error.

I'll look into this, maybe this is the real problem!

Thanks

gregbrowndev · 2022-08-18T14:58:38Z

It doesn't look specifically like a httpx issue. This test passes (calling out to a real http API):

import asyncio

import httpx
import pytest
from httpx import AsyncClient


async def send_request(client: AsyncClient, url: str) -> dict:
    r = await client.get(url)
    return r.json()


@pytest.fixture
def httpx_client() -> AsyncClient:
    return httpx.AsyncClient()


def test_async(httpx_client: AsyncClient):
    url = "https://dummyjson.com/products/1"

    loop = asyncio.get_event_loop()
    coroutine = send_request(httpx_client, url)
    actual = loop.run_until_complete(coroutine)

    assert "id" in actual

However, if you add the decorator, you get the error below (different to the original one)

Stacktrace

Traceback (most recent call last):
  File "/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/_base.py", line 330, in _invoke_callbacks
    callback(self)
  File "/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/asyncio/futures.py", line 398, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/asyncio/base_events.py", line 801, in call_soon_threadsafe
    self._write_to_self()
  File "/Users/gregorybrown/.pyenv/versions/3.10.4/lib/python3.10/asyncio/selector_events.py", line 135, in _write_to_self
    csock.send(b'\0')
  File "/Users/gregorybrown/Library/Caches/pypoetry/virtualenvs/tariff-management-6yv2RoDp-py3.10/lib/python3.10/site-packages/mocket/mocket.py", line 391, in send
    entry = self.get_entry(data)
  File "/Users/gregorybrown/Library/Caches/pypoetry/virtualenvs/tariff-management-6yv2RoDp-py3.10/lib/python3.10/site-packages/mocket/mocket.py", line 258, in get_entry
    return Mocket.get_entry(self._host, self._port, data)
  File "/Users/gregorybrown/Library/Caches/pypoetry/virtualenvs/tariff-management-6yv2RoDp-py3.10/lib/python3.10/site-packages/mocket/mocket.py", line 430, in get_entry
    host = host or Mocket._address[0]
AttributeError: type object 'Mocket' has no attribute '_address'

mindflayer · 2022-08-18T14:59:28Z

This looks like a mocket one! :) I'll have a look at it ASAP.

mindflayer · 2022-08-18T17:56:20Z

First of all, the error related to JSON decoding is a symptom of something wrong. It looks like mocket is not intercepting - or serving - the call. You can see it from the following screenshot:

It's exactly the content of the homepage of the website it was supposed to mock (example.org). When you write snippets like that I suggest you to use fake URLs.
Now I'm trying to understand why the client from the fixture does not work properly.

mindflayer · 2022-08-18T18:36:47Z

It looks like the client from the fixture is living in a non-perfectly-mocked reality.
Mocket is doing right but, for some reason, after it serves the response, a real call happens.

Like you said, the fix is in enabling mocket from inside the fixture like I did, or using the decorator like you mentioned before.

import json

import httpx
import pytest

from mocket.mockhttp import Entry
from mocket import Mocketizer


@pytest.fixture
def httpx_client() -> httpx.AsyncClient:
    with Mocketizer():
        yield httpx.AsyncClient()


@pytest.mark.asyncio
async def test_httpx(httpx_client):
    url = "https://foo.bar/"
    data = {"message": "Hello"}

    Entry.single_register(
        Entry.GET,
        url,
        body=json.dumps(data),
        headers={"content-type": "application/json"},
    )

    async with httpx_client as client:
        response = await client.get(url)

        assert response.json() == data

mindflayer · 2022-08-18T21:04:14Z

I've just added the above test to mocket's test suite. You can have a look at the linked PR.

mindflayer mentioned this issue Aug 18, 2022

Improving tests for httpx #186

Merged

mindflayer closed this as completed Aug 18, 2022

mindflayer added the question label Aug 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot inject HTTPX client as pytest fixture #185

Cannot inject HTTPX client as pytest fixture #185

gregbrowndev commented Aug 18, 2022

mindflayer commented Aug 18, 2022 •

edited

mindflayer commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

mindflayer commented Aug 18, 2022

mindflayer commented Aug 18, 2022 •

edited by gsalluzzo-tc

mindflayer commented Aug 18, 2022 •

edited

mindflayer commented Aug 18, 2022 •

edited by gsalluzzo-tc

Cannot inject HTTPX client as pytest fixture #185

Cannot inject HTTPX client as pytest fixture #185

Comments

gregbrowndev commented Aug 18, 2022

mindflayer commented Aug 18, 2022 • edited

mindflayer commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

gregbrowndev commented Aug 18, 2022

mindflayer commented Aug 18, 2022

mindflayer commented Aug 18, 2022 • edited by gsalluzzo-tc

mindflayer commented Aug 18, 2022 • edited

mindflayer commented Aug 18, 2022 • edited by gsalluzzo-tc

mindflayer commented Aug 18, 2022 •

edited

mindflayer commented Aug 18, 2022 •

edited by gsalluzzo-tc

mindflayer commented Aug 18, 2022 •

edited

mindflayer commented Aug 18, 2022 •

edited by gsalluzzo-tc