Repository navigation
Raw deflate decoding fails when the first response chunk is one byte #3799
Replies: 2 comments
|
Independently reproduced on Linux with Python 3.12.14 and zlib 1.3.1.zlib-ng, using HTTPX master I checked 48 in-memory response cases: raw and zlib-wrapped deflate, empty / 8-byte / 2048-byte bodies, chunk sizes 1 / 2 / 3 / whole stream, synchronous reads and asynchronous reads under asyncio.
This adds a Linux/zlib-ng check to your macOS results and confirms the behavior is not tied to a live HTTP server. I have not run the full HTTPX suite or Trio in this environment, so this is focused reproduction evidence rather than full approval of the patch. |
|
I reckon, this is a real bug in the current DeflateDecoder. The decoder starts with The bug is exactly what you described: when the first raw-deflate chunk is a single byte, the default zlib.decompressobj()`does not yet have enough input to reject the (missing) zlib header, so it returns an empty result without raising. Because the call did not raise, first_attempt is already False. When the second chunk arrives, the default decompressor finally raises incorrect header check, but the fallback to raw deflate is no longer allowed, so DecodingError is raised. Your proposed fix: retaining the initial bytes until the header can be judged would address this without changing the normal single-chunk path. |
Uh oh!
There was an error while loading. Please reload this page.
A raw-deflate response decodes correctly when supplied as one chunk, but raises
DecodingErrorwhen the first chunk contains just one byte. I reproduced this on current master (b5addb64f0161ff6bfe94c124ef76f6a1fba5254).The raw-deflate fallback is already supported by
DeflateDecoder. With a one-byte first chunk, zlib has not seen enough input to reject the header, butfirst_attemptis set toFalse. The second chunk then raises without trying the raw decoder. This also reproduces withResponse.aread(), under both asyncio and Trio; no server or network request is needed.I prepared a small candidate fix and regression tests: retain the initial byte until both header bytes are available, then keep the existing fallback. Tests cover raw deflate and zlib-wrapped deflate with small synchronous chunks and one-byte async chunks.
The decoder tests, formatting, typing and lint checks pass. The full suite has one existing
test_write_timeout[trio]resource-warning failure, which I also reproduced with the original decoder. Excluding that single test leaves 1424 passed and 1 skipped.Environment: macOS 26.6.2 arm64, Python 3.13.15, HTTPX 0.28.1 checkout, zlib 1.2.12.
Is this approach suitable for a PR? I'm raising it here first as requested by the contribution guide.
All reactions