HttpClient deflate decompression error

[FeeSaver]

Hi, I just tried to move project from Net 5 to latest preview of Net 6 and I have noticed issue with automatic decompression.
Gzip and Brotli seems to work fine, but Deflate is throwing IO error with invalid stream data.
When I change back to Net 5, everything works fine again. Here is part of code that throws the error on last line with stream.ReadAsync. I think this happen only when the website is using https and I receive response compressed with Deflate, also the HttpCompletionOption.ResponseHeadersRead might be causing some issue.

var _httpClientHandler = new SocketsHttpHandler
{
    AutomaticDecompression = DecompressionMethods.All
};
var _client = new HttpClient(_httpClientHandler, false);	
var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead,)
if (response.IsSuccessStatusCode)
{
    Stream contentStream = await response.Content.ReadAsStreamAsync(_token);
    if (contentStream != Stream.Null)
    {
        int readBytes = 0;
        var rentedBuffer = pool.Rent(64 * 1024);
        var bufferMemory = new Memory<byte>(rentedBuffer, 0, rentedBuffer.Length);
        int readChunk = await stream.ReadAsync(bufferMemory.Slice(readBytes));
    }
}

System.IO.InvalidDataException: The archive entry was compressed using an unsupported compression method.
   at ErrorCode System.IO.Compression.Inflater.Inflate(FlushCode flushCode)
   at ErrorCode System.IO.Compression.Inflater.ReadInflateOutput(Byte* bufPtr, int length, FlushCode flushCode, out int bytesRead)
   at void System.IO.Compression.Inflater.ReadOutput(Byte* bufPtr, int length, out int bytesRead)
   at int System.IO.Compression.Inflater.InflateVerified(Byte* bufPtr, int length)
   at async ValueTask<int> System.IO.Compression.DeflateStream.ReadAsyncMemory(Memory<byte> buffer, CancellationToken cancellationToken)+Core(?)

[KalleOlaviNiemitalo]

Might be caused by #42717. If I understand correctly, .NET used to treat Content-Encoding: deflate as only the RFC 1951 "deflate" compression, but RFC 2616 says it requires also the RFC 1950 zlib header. This was fixed in .NET, but now if the HTTP server violates RFC 2616 by omitting the zlib header, it won't work.

[FeeSaver]

Is there any planned workaround? There are website apis that work for me using any browser or basically anything other than Net6 app. Or just half of websites using deflate wont work from now on? I have seen this issue on many application that use NodeJS. I looked at what they use and they have 2 versions of deflate available for almost 10 years.

Class: zlib.Deflate#
Added in: v0.5.8
Compress data using deflate.

Class: zlib.DeflateRaw#
Added in: v0.5.8
Compress data using deflate, and do not append a zlib header.

https://nodejs.org/api/zlib.html#zlib_class_zlib_deflate

[KalleOlaviNiemitalo]

.NET has DeflateStream and ZlibStream. I imagine you could work around the issue by disabling automatic decompression, which now uses ZlibStream for "deflate", and instead adding the Accept-Encoding header field in your app and using DeflateStream if the server chooses "deflate". Or just do not request "deflate" from the server.

[KalleOlaviNiemitalo]

If there is no code fix done to automatically detect noncompliant servers and decode the plain DEFLATE without zlib, I think this should be documented as a breaking change.

[FeeSaver]

I am hoping for some fix really, .Net deflate worked well "incorrectly" for years, but atleast it worked. Now it is correct to latest standard, but not useable. I guess deflate is used mostly on older servers, the newer servers that maybe support the newer standard use gzip or brotli anyway. My problem is that some sites just return deflate content as default whether I set it as accepted encoding or not, or maybe .Net add the deflate as default acepted encoding, I didnt check.
I guess they just assume its suppored by everything. I hope this gets changed in future, I will keep using Net5 for now.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hi, I just tried to move project from Net 5 to latest preview of Net 6 and I have noticed issue with automatic decompression.
Gzip and Brotli seems to work fine, but Deflate is throwing IO error with invalid stream data.
When I change back to Net 5, everything works fine again. Here is part of code that throws the error on last line with stream.ReadAsync. I think this happen only when the website is using https and I receive response compressed with Deflate, also the HttpCompletionOption.ResponseHeadersRead might be causing some issue.

var _httpClientHandler = new SocketsHttpHandler { AutomaticDecompression = DecompressionMethods.All }; var _client = new HttpClient(_httpClientHandler, false); var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead,) if (response.IsSuccessStatusCode) { Stream contentStream = await response.Content.ReadAsStreamAsync(_token); if (contentStream != Stream.Null) { int readBytes = 0; var rentedBuffer = pool.Rent(64 * 1024); var bufferMemory = new Memory<byte>(rentedBuffer, 0, rentedBuffer.Length); int readChunk = await stream.ReadAsync(bufferMemory.Slice(readBytes)); } }

System.IO.InvalidDataException: The archive entry was compressed using an unsupported compression method. at ErrorCode System.IO.Compression.Inflater.Inflate(FlushCode flushCode) at ErrorCode System.IO.Compression.Inflater.ReadInflateOutput(Byte* bufPtr, int length, FlushCode flushCode, out int bytesRead) at void System.IO.Compression.Inflater.ReadOutput(Byte* bufPtr, int length, out int bytesRead) at int System.IO.Compression.Inflater.InflateVerified(Byte* bufPtr, int length) at async ValueTask<int> System.IO.Compression.DeflateStream.ReadAsyncMemory(Memory<byte> buffer, CancellationToken cancellationToken)+Core(?)

Author:	FeeSaver
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[stephentoub]

@karelz, this is the issue we were discussing offline.

[KalleOlaviNiemitalo]

Are you planning an AppContext switch?