ByteArrayHttpMessageConverter truncation when content is compressed

[deathy]

Seems that ByteArrayHttpMessageConverter may truncate response body when the response is compressed (via gzip,deflate,etc.)

Context:

• Spring Boot 4.0.1
• HttpComponents 5.6
  • which as per https://hc.apache.org/news.html 5.6 introduced "Support for pluggable content codecs via Commons-Compress in the classic transport. (optional)." and "Support for transparent content decompression and content compression with deflate, gzip, zstd (optional), and brotli (optional) codecs in the async transport. "
• RestClient reading external URL that is served with gzip compression
• .retrieve().body(byte[].class) only returns size of compressed content not entire decompressed content

From debugging session:

Response headers:

• Content-Length: 9075
• Content-Encoding: gzip

From ByteArrayHttpMessageConverter:

@Override
public byte[] readInternal(Class<? extends byte[]> clazz, HttpInputMessage message) throws IOException {
	long length = message.getHeaders().getContentLength();
	return (length >= 0 && length < Integer.MAX_VALUE ?
			message.getBody().readNBytes((int) length) : message.getBody().readAllBytes());
}

So in case of existing Content-Length header is only reads that many bytes giving back 9075 bytes of decompressed content.
But message.getBody().readAllBytes()) returns proper byte array of length 56815 bytes.

The check there seems to assume plain not-compressed content, since Content-Length is only about the representation (number of gzipped bytes) not actual response body bytes after any decompression.

There's the edge-cases of content-length header not being present meaning -1 or larger than Integer.MAX_VALUE but in both those cases readAllBytes() will be called. Why not always call readAllBytes() ? (range request?)

[bclozel]

I think you have uncovered a bug in the compression support in the client itself.

Support for transparent content decompression and content compression with deflate, gzip, zstd (optional), and brotli (optional) codecs in the async transport.

In this case, I think the client should remove/update the Content-Length header, otherwise (de)compression is not transparent at all. I guess this support already adds the "Content-Encoding" header on the way out? From an application perspective, an HTTP message that has Content-Length: 9075, Content-Encoding: gzip and a decompressed response body is plain wrong.

Please raise this issue with the library maintainers. If you believe my analysis is wrong, please provide a minimal sample application so that I can reopen this issue and have a look at it. Thanks!

[deathy]

Thanks for the analysis. For reference I opened up HttpComponents issue https://issues.apache.org/jira/browse/HTTPCLIENT-2409

Wasn't quite clear on it before since the InputStream contract was preserved meaning all bytes were correctly read if reading fully. But I checked both JDK HttpClient and Jetty HttpClient and general behavior is removing Content-Length header if decompression is happening, so that's common expected behavior.
Which upon further study was also behavior in older HttpComponents versions just seems to have been broken with the most recent changes.