Got request is silently failing on compressed data when an interceptor is set up

[OlivierRiberi]

Hi,

I'm using Got and it seems like the request is silently failing while fetching compressed data (like brotli or gzip).
In the request option, Got provides a decompress option (default: true) that use decompress-response internally.
Of course, everything is working fine when the interceptor is not set up.

Thanks

Repro: https://replit.com/@qyfjs9bfwd/Interceptor-and-Got-with-compressed-data

import got from 'got'

import { BatchInterceptor } from '@mswjs/interceptors'
import { ClientRequestInterceptor } from '@mswjs/interceptors/lib/interceptors/ClientRequest/index.js'
import { XMLHttpRequestInterceptor } from '@mswjs/interceptors/lib/interceptors/XMLHttpRequest/index.js'

const interceptor = new BatchInterceptor({
  name: 'my-interceptor',
  interceptors: [
    new ClientRequestInterceptor(),
    new XMLHttpRequestInterceptor(),
  ]
})

interceptor.apply()

interceptor.on('request', (request) => {
  console.log('interceptor listener')
})

// Working
const resp = await got('https://httpbin.org/brotli', { decompress: false })
console.log(resp.body)

// Not working
const resp2 = await got('https://httpbin.org/brotli')
console.log(resp2.body) // Does not show up

// Both requests are working if interceptor.apply() is commented out

[blaenk]

I was about to create a new ticket but I think I'm running into this as well.

I am not using Got, but a site's responses have e.g. Content-Encoding: br or gzip or deflate and I get all kinds of exceptions thrown like:

Uncaught RangeError: Failed to construct 'TextDecoder': The encoding label provided ('br') is invalid.
Uncaught RangeError: Failed to construct 'TextDecoder': The encoding label provided ('gzip') is invalid.

And so on.

While searching around to figure out what the issue might be, it seems like it might be possible to decompress/compress also with native browser APIs using DecompressionStream and CompressionStream:

https://gist.github.com/Explosion-Scratch/357c2eebd8254f8ea5548b0e6ac7a61b

function compress(string, encoding) {
  const byteArray = new TextEncoder().encode(string);
  const cs = new CompressionStream(encoding);
  const writer = cs.writable.getWriter();
  writer.write(byteArray);
  writer.close();
  return new Response(cs.readable).arrayBuffer();
}

function decompress(byteArray, encoding) {
  const cs = new DecompressionStream(encoding);
  const writer = cs.writable.getWriter();
  writer.write(byteArray);
  writer.close();
  return new Response(cs.readable).arrayBuffer().then(function (arrayBuffer) {
    return new TextDecoder().decode(arrayBuffer);
  });
}

[blaenk]

I was investigating some more and I noticed that simply passing undefined to decodeBuffer() instead of the strings br, gzip, etc., worked perfectly fine. This is on browser by the way, so maybe different from this issue after all.

It actually seems like using the Content-Encoding header and passing it straight through to TextEncoder doesn't quite make sense because Content-Encoding is typically about compression, whereas TextEncoder's parameter is about character encoding like utf8.

I'm gonna offer a PR to remove that feature.

[kettanaito]

@blaenk, you're right on this. We are mistakingly forwarding response content encoding to the text decoder of buffer bodies in XMLHttpRequest. Thanks for opening a fix in #318!

Since the original issue is related to compression, I believe that your investigation and the fix will close this issue.

[kettanaito]

Released: v0.19.2 🎉

This has been released in v0.19.2!

• 📄 Release notes
• 📦 npm package

Make sure to always update to the latest version (npm i @mswjs/interceptors@latest) to get the newest features and bug fixes.

---

Predictable release automation by @ossjs/release.

[OlivierRiberi]

Thank you for the investigation!
Unfortunately, I ran again my repro and it seems like the issue still persists with the latest version: 0.19.4
I wish I could be more helpful but I'm not familiar with the internals of @mswjs/interceptors nor got.