"Accept-Encoding" inserted into http requests headers

[iidebyo]

What version of Bun is running?

1.0.0+822a00c4d508b54f650933a73ca5f4a3af9a7983

What platform is your computer?

Linux 5.15.90.1-microsoft-standard-WSL2 x86_64 x86_64

What steps can reproduce the bug?

import h from 'node:http'
import z from 'node:zlib'
h.get('http://example.com/', async r => {
  var d = []
  var f = () => {
    console.log(z.gunzipSync(Buffer.concat(d)).toString())
    console.log(Buffer.concat(d).length)
  }
  for await (const c of r) {
    d.push(c)
  }
  f()
})

What is the expected behavior?

An error should appear such as: Error: incorrect header check.

What do you see instead?

html from http://example.com

Additional information

bun/src/http_client_async.zig

Lines 2125 to 2128 in 8615b8a

	if (!override_accept_encoding) {
	request_headers_buf[header_count] = accept_encoding_header;
	header_count += 1;
	}

[Axiean]

The behavior you are observing is related to how the zlib.gunzipSync method is used in your code and how it handles the response from the HTTP request.

In your code, you are making an HTTP request to http://example.com/ and attempting to decode the response using zlib.gunzipSync. However, this method is synchronous and expects the entire compressed data to be available before it can be decompressed.

When you use for await (const c of r) to iterate over the response data stream, you are processing the response incrementally, but zlib.gunzipSync doesn't support this incremental processing. It expects the complete compressed data in one go.

As a result, when you call zlib.gunzipSync(Buffer.concat(d)), it attempts to decompress the concatenated data, which may not be a complete gzip stream, leading to the behavior you described.

To handle compressed responses from an HTTP request in a more appropriate manner, you should use asynchronous decompression and handle the response incrementally. You can use the zlib.gunzip function from the 'zlib' module instead of zlib.gunzipSync. Here's an example of how you can modify your code:

import http from 'node:http';
import zlib from 'node:zlib';

const req = http.get('http://example.com/', (res) => {
  const gunzip = zlib.createGunzip();
  res.pipe(gunzip);

  let data = '';

  gunzip.on('data', (chunk) => {
    data += chunk;
  });

  gunzip.on('end', () => {
    console.log(data);
    console.log(data.length);
  });
});

req.end();

In this example, we create a gunzip stream using zlib.createGunzip() and pipe the response data through it. The data is then accumulated incrementally in the data variable. This approach allows for proper handling of gzip-compressed responses.

[iidebyo]

Please take another look at expected and observed behavior.

[iidebyo]

fixed by #5057