--compressed fails on multi-member (concatenated) gzip responses

[HwangRock]

I did this

I was fetching large compressed log archives over HTTP with curl --compressed.
Some of these are made by concatenating several gzip files, which is a common shape for rotated logs and what pigz or a plain cat a.gz b.gz produce.
Per RFC 1952 section 2.2 that is still one valid gzip stream (gunzip and zcat decode it fine), but curl stops partway through.

curl decodes only the first gzip member and then fails the transfer with CURLE_WRITE_ERROR (23), dropping the rest of the data.

Minimal reproduction, no real server needed:

import zlib
data = b"HTTP/1.1 200 OK\r\nContent-Encoding: gzip\r\nConnection: close\r\n\r\n"
def member(s):
    c = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
    return c.compress(s) + c.flush()
data += member(b"foo") + member(b"bar")   # two concatenated gzip members
open("foobar.bin", "wb").write(data)

# terminal 1
while true; do nc -l 127.0.0.1 8000 < foobar.bin; done
# terminal 2
curl -sv --compressed -o out http://127.0.0.1:8000/

out ends up with foo and curl exits with code 23. gunzip < foobar.bin
returns foobar.

I expected the following

out to contain foobar and the transfer to succeed, the same way gunzip, zcat
and gzip.decompress() in Python handle it.

Notes

This came up before in #3796, which went stale without a conclusion.
That thread weighed two options: silently ignore everything after the first member (what some browsers do), or decode every member. The argument against silently ignoring was data integrity, since curl is usually an automated tool and quietly losing data
is easy to miss.

Decoding all members keeps that integrity and matches the reference tools.
The spot is process_trailer() in lib/content_encoding.c: once zlib returns Z_STREAM_END for the first member, the remaining bytes (the next member, starting with the gzip magic 1f 8b) are taken as unexpected trailing data and rejected. The usual zlib idiom is to call inflateReset() when Z_STREAM_END is hit with input still pending and continue.

I have a patch and a test and can send a PR if this direction looks right.

curl/libcurl version

Reproduced with curl 8.7.1 (zlib/1.2.12). The relevant code in lib/content_encoding.c is unchanged on current master.

operating system

macOS, but it is not platform specific.

[vigneshwarrvenkat]

I can reproduce this with the Python reproducer in the issue description: concatenated gzip members decode only the first member, then the transfer fails with CURLE_WRITE_ERROR (23).
Root cause
In lib/content_encoding.c, when inflate() returns Z_STREAM_END for gzip, the code goes through process_trailer(), which treats any remaining input as an error and calls exit_zlib(). That is correct for a single-member stream, but RFC 1952 section 2.2 allows a gzip stream to consist of several concatenated members (common for rotated logs / pigz output).
Two failure modes follow from this:

1. If the next member is already in the buffer when the first member ends, process_trailer() sees avail_in > 0 and returns CURLE_WRITE_ERROR.
2. If the first member ends before the next chunk arrives, exit_zlib() runs early, the next gzip_do_write() then hits ZLIB_UNINIT and also fails.
   Proposed fix
   Handle member boundaries in gzip transparent mode (ZLIB_INIT_GZIP) separately from deflate/raw-deflate:

• On Z_STREAM_END, if avail_in >= 2 and the next bytes are gzip magic (0x1f 0x8b), call inflateReset2(z, MAX_WBITS + 32) and continue inflating the next member in the same loop.
• If avail_in == 0, reset the inflate state but do not call exit_zlib() yet more body data may arrive in a later write. Final cleanup stays in gzip_do_close().
• If avail_in > 0 but the bytes are not a gzip header, treat that as trailing garbage and fail.
  Leave process_trailer() unchanged for non-gzip paths.
  Test
  Add an integration test (tests/data/test2217) with two concatenated gzip members (foo + bar) and verify --compressed outputs foobar.
  I have a patch ready locally and plan to open a PR shortly. Happy to adjust if maintainers prefer a different approach.

[HwangRock]

@vigneshwarrvenkat Thank you for verifying the issue and sharing a detailed analysis!
I originally opened this issue because I wanted to discuss the direction with the maintainers first.
As I noted at the end of the description, I actually have a patch and an integration test ready locally.
I will go ahead and open the PR shortly so we can review the implementation together.
Thanks again for your insights!

[icing]

Hmm, interesting. My read of HTTP Content-Encoding is that, whatever the coding, it is applied to the URL "resource". Singular. And the type of that resource is described in the Content-Type header. Therefore, encodings that contain multiple files are abusing the HTTP design here. For example, tar is not a valid HTTP Content-Encoding either.

Imagine a client that does not support gzip. The server is the expected to send the resource content without any content encoding. Raw. Would that be the bytes of the multiple files just concatenated? Hardly useful.

And therefore, sending a gzip with multiple entries is not correct in HTTP. It's a server misconfiguration. I think such a response should fail. Browsers have different priorities than curl.

I dare even say: whoever cooked up those "reference tools" that extract multiple files did a hack, not understanding or ignoring the HTTP protocol.

Update: what I do not understand here: why not simply send the gzip without Content-Encoding?

[HwangRock]

@icing That's an interesting question, and I think it comes down to what the client actually wants.

If it's a gzip file meant to be saved, then yeah — application/gzip with no Content-Encoding, and putting Content-Encoding on it is just the server being wrong. No disagreement there, that's the S3/CloudTrail kind of case.

The one I keep running into is the opposite: the client ran curl --compressed (or sent Accept-Encoding: gzip) and wants the decoded body, and the server just gzipped a response, some JSON or a log stream, for transit. Handing back application/gzip there gives the caller a blob to unpack themselves and ignores the encoding they asked for. The multiple members often aren't deliberate either — they fall out of the server gzipping the response chunk by chunk.

And unpacking it isn't really exotic. RFC 9110 8.4.1.3 points to RFC 1952 for the gzip coding, and RFC 1952 2.2 describes a gzip file as a series of members one after another, so a single member was never required. It is split in practice: browsers stop at the first member (the WHATWG Compression Streams spec even limits gzip to one), and reqwest and OkHttp stop or throw — but Go's net/http and Python's requests both read every member. You said browsers have different priorities than curl, and the runtimes that do share curl's priorities are the ones reading all of it. Often the server isn't mine to fix anyway (Prometheus, an S3 bucket), so telling it to behave isn't really on the table.

If curl would still rather call it malformed and fail, that's fair — maybe with a more specific error than CURLE_WRITE_ERROR, so it's at least distinguishable from a truncated body.

[icing]

The multiple members often aren't deliberate either — they fall out of the server gzipping the response chunk by chunk.

The idea of content encoding is really that it "wraps" the bytes stream of the plain, equivalent response. Some servers do that "on the fly", some support serving it from pre-computed files to save cpu. It sounds as if some software generated gzip with multiple entries and then tells the server that this is the content-encoded file to serve. Which is not how servers and HTTP intended it.

But sure, people do weird things and unless you tell curl to decompress, it will work seemingly well enough.

I agree with you that curl could make more effort in the error message. When it sees there is another entry after the first, it could tell the user why the decoding failed. That'd be helpful.

[HwangRock]

@icing Sounds good! I'll have curl spot the extra member and fail with a message that explains it hit a second gzip member, rather than the bare write error. I'll open a PR for that.