Error: Receive buffer too long - Response header greater than 16kB

[DHuyAu]

Hi,

I have recently replace requests with httpx and have encountered an issue when the server responds with a large response header.
I believe it is to do with the h11 connection that is created when the request is issued.
https://github.com/python-hyper/h11/blob/master/h11/_connection.py#L473-L478

I was wondering if the httpx client created can be configurable similar to what was done in uvicorn: encode/uvicorn#1514

I was able to get around this by creating a custom transport similar to the default HTTPTransport and setting the H11 Connection with a greater size. I have attached the custom transport as a reference.

I had tried to patch h11 DEFAULT_MAX_INCOMPLETE_EVENT_SIZE prior to import httpx to no avail.

import contextlib
from typing import Iterable, Iterator

import httpx
import httpcore
import h11


HTTPCORE_EXC_MAP = {
    httpcore.TimeoutException: httpx.TimeoutException,
    httpcore.ConnectTimeout: httpx.ConnectTimeout,
    httpcore.ReadTimeout: httpx.ReadTimeout,
    httpcore.WriteTimeout: httpx.WriteTimeout,
    httpcore.PoolTimeout: httpx.PoolTimeout,
    httpcore.NetworkError: httpx.NetworkError,
    httpcore.ConnectError: httpx.ConnectError,
    httpcore.ReadError: httpx.ReadError,
    httpcore.WriteError: httpx.WriteError,
    httpcore.ProxyError: httpx.ProxyError,
    httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
    httpcore.ProtocolError: httpx.ProtocolError,
    httpcore.LocalProtocolError: httpx.LocalProtocolError,
    httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
}

@contextlib.contextmanager
def map_httpcore_exceptions() -> Iterator[None]:
    try:
        yield
    except Exception as exc:  # noqa: PIE-786
        mapped_exc = None

        for from_exc, to_exc in HTTPCORE_EXC_MAP.items():
            if not isinstance(exc, from_exc):
                continue
            # We want to map to the most specific exception we can find.
            # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to
            # `httpx.ReadTimeout`, not just `httpx.TimeoutException`.
            if mapped_exc is None or issubclass(to_exc, mapped_exc):
                mapped_exc = to_exc

        if mapped_exc is None:  # pragma: nocover
            raise

        message = str(exc)
        raise mapped_exc(message) from exc


class CustomResponseStream(httpx.SyncByteStream):
    def __init__(self, httpcore_stream: Iterable[bytes]):
        self._httpcore_stream = httpcore_stream

    def __iter__(self) -> Iterator[bytes]:
        with map_httpcore_exceptions():
            for part in self._httpcore_stream:
                yield part

    def close(self) -> None:
        if hasattr(self._httpcore_stream, "close"):
            self._httpcore_stream.close()  # type: ignore


class CustomHTTP11Connection(httpcore.HTTP11Connection):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._h11_state = h11.Connection(our_role=h11.CLIENT, max_incomplete_event_size=100000000)


class CustomResponseStream(httpx.SyncByteStream):
    def __init__(self, httpcore_stream: Iterable[bytes]):
        self._httpcore_stream = httpcore_stream

    def __iter__(self) -> Iterator[bytes]:
        with map_httpcore_exceptions():
            for part in self._httpcore_stream:
                yield part

    def close(self) -> None:
        if hasattr(self._httpcore_stream, "close"):
            self._httpcore_stream.close()  # type: ignore


class CustomHttpConnection(httpcore.HTTPConnection):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def handle_request(self, request: httpcore.Request) -> httpcore.Response:
        if not self.can_handle_request(request.url.origin):

            raise RuntimeError(
                f"Attempted to send request to {request.url.origin} on connection to {self._origin}"
            )

        with self._request_lock:
            if self._connection is None:
                try:
                    stream = self._connect(request)

                    ssl_object = stream.get_extra_info("ssl_object")
                    http2_negotiated = (
                            ssl_object is not None
                            and ssl_object.selected_alpn_protocol() == "h2"
                    )
                    if http2_negotiated or (self._http2 and not self._http1):
                        from httpcore import HTTP2Connection

                        self._connection = HTTP2Connection(
                            origin=self._origin,
                            stream=stream,
                            keepalive_expiry=self._keepalive_expiry,
                        )
                    else:
                        self._connection = CustomHTTP11Connection(
                            origin=self._origin,
                            stream=stream,
                            keepalive_expiry=self._keepalive_expiry,
                        )
                except Exception as exc:
                    self._connect_failed = True
                    raise exc
            elif not self._connection.is_available():
                raise httpcore.ConnectionNotAvailable()

        return self._connection.handle_request(request)


class CustomHTTPTransport(httpx.BaseTransport):
    def __init__(
        self, *args, **kwargs
    ) -> None:
        super().__init__(*args, **kwargs)

    def handle_request(
        self,
        request: httpcore.Request,
    ) -> httpcore.Response:
        assert isinstance(request.stream, httpx.SyncByteStream)

        req = httpcore.Request(
            method=request.method,
            url=httpcore.URL(
                scheme=request.url.raw_scheme,
                host=request.url.raw_host,
                port=request.url.port,
                target=request.url.raw_path,
            ),
            headers=request.headers.raw,
            content=request.stream,
            extensions=request.extensions,
        )
        with map_httpcore_exceptions():
            response = CustomHttpConnection(
                req.url.origin
            ).handle_request(req)

        assert isinstance(response.stream, Iterable)

        return httpx.Response(
            status_code=response.status,
            headers=response.headers,
            stream=CustomResponseStream(response.stream),
            extensions=response.extensions,
        )


if __name__ == "__main__":
    # This succeeds with a response that had header >16kB
    client = httpx.Client(transport=CustomHTTPTransport())
    resp = client.get("http://localhost:8000/test-httpx/")
    print(resp.headers)

    # This failed with a response that had header >16kB
    resp = httpx.get("http://localhost:8000/test-httpx/")
    print(resp.headers)

[tomchristie]

Yup, I think we could increase the max_incomplete_event_size that we use.
Having the same value that curl uses would be reasonable.

Their limit is 1024 * 100.
Our limit is currently 1024 * 16.

We can either fix this by changing the default in h11, or by changing the configuration in httpcore, which can accept max_incomplete_event_size=... as you've done in this example.

Which of those two approaches do we think is more reasonable?

[zanieb]

It seems simplest to adjust the default to a new reasonable value then consider adding configuration in the future if there is another request.

[tomchristie]

Let me rephrase that...

I don't want httpx to allow any configuration of what value we use here.
Going with the same value that curl uses will be exactly what we'd like.

But we do need to either change the default value in the h11 package, here...

https://github.com/python-hyper/h11/blob/95cd3fa36e73b9b9c55e4d97b4c8fdee1c60b498/h11/_connection.py#L58

From...

DEFAULT_MAX_INCOMPLETE_EVENT_SIZE = 16 * 1024

To...

DEFAULT_MAX_INCOMPLETE_EVENT_SIZE = 100 * 1024

Or leave the h11 default as it is, but change our configuration in httpcore in order to override it. Which we could do by switching this line here...

https://github.com/encode/httpcore/blob/d6a96adcef7344247dab39591e3edec5f9b6713d/httpcore/_async/http11.py#L60

From...

self._h11_state = h11.Connection(our_role=h11.CLIENT)

To...

self._h11_state = h11.Connection(our_role=h11.CLIENT, max_incomplete_event_size=100 * 1024)

I suppose the best first pass would be to raise a ticket with h11 suggesting that the limit be raised in line with curl?

If that gets accepted on h11, then great. If it doesn't get accepted on h11, then that's also perfectly okay. In that case we would then choose to override it specifically for httpcore/httpx.

[zanieb]

Ah I see, I misread your initial comment — thanks for the clarification! I totally agree.

You may want to override it in httpcore anyway for compatibility with a wider range range of h11 versions, but I'm not sure how much of a priority that is.

[DHuyAu]

I think the second approach is more reasonable.

I don't believe requesting changes at h11 is the correct place as the comments suggests the limits were placed intentionally with h11 being used in both client and server applications.
https://github.com/python-hyper/h11/blob/master/h11/_connection.py#L53-L58

In regards to configurability, is this related to this post: encode/uvicorn#1234 (comment)? You don't want configuration littered throughout the code to reduce overhead when reading code?

[tomchristie]

Okay.

I'm okay either with someone making a pull request to httpcore, setting max_incomplete_event_size=100 * 1024 in both _async/h11.py and _sync/h11.py, or else with someone raising this as an issue on h11, and seeing what their maintainers prefer here. Either way is okay for us.

You don't want configuration littered throughout the code to reduce overhead when reading code?

Adding too much configurability just adds complexity and noise to the API.