How to decompress the response?

[psytechno604]

Just a very simple example, almost exactly as in docs:

const { Writable } = require('stream')

const client = new Client(`http://ad.service.serverside.ai`)
const bufs = [];

 (async () => {
    try {
      await client.stream({
        path: '/api/ad-server-static_2.xml',
        method: 'GET',
        opaque: { bufs }
      }, ({ statusCode, headers, opaque: { bufs } }) => {
        console.log(`response received ${statusCode}`)
        console.log('headers', headers)
        return new Writable({
          write (chunk, encoding, callback) {
            bufs.push(chunk)
            callback()
          }
        })
      })

      console.log(Buffer.concat(bufs).toString('utf-8'))

      client.close()
    } catch (error) {
      console.error(error)
    }
})()

The URL http://ad.service.serverside.ai/api/ad-server-static_2.xml actually returns XML, but in the output there is something strange:

���o�0���WxyO�t�
.ik�<>���Mpyx�u�Dn��>���z���,W\kW���K���滴��F,�.-��&ʡUgq&�)дdžl���:>i��Y�����b�TUME�sEmC��y�iC���z/��T$Ϻ��_�#E�gX����1���'����p�5�� �HP��r%����7̸G�m���.�UKp͙�W"�/E�\��Һ�@񉽳w�Y�0�E�۟�'Y
f�]�K>�b25�ʧ��5��5�"�6R� ��R��%Rqo)2&��˸����i�j9W �o���_ƣ�����gn��mu��O��� ?Z�������gq��i����5s8�? ��&�ݸ�Z�/�2�\���j�+�m�.m���]�7�X8�}9�L]����1]P�ҧ*Z�ء�eW���z�σۻ����]9��:�9�xRa+�˟oe����ځ�` ��pF8#��6'��pF8#�g�3��pF8#�g�3��pF8#�Λ8�3��lg�3��L�;g�3��pF8#�ί�A8#�7r��pF8#���εHd3����l��
��7���-�>

If it is because of gzip compression - how to properly decompress it?

[ronag]

Pipe through a decompressor.

[psytechno604]

Pipe through a decompressor.

could you please give an example?

[psytechno604]

all my efforts ended with "incorrect header check"

[psytechno604]

isn't undici.fetch experimental, with possible breaking changes?

This is experimental and is not yet fully compliant with the Fetch Standard. We plan to ship breaking changes to this feature until it is out of experimental.

https://npm.devtool.tech/undici

I tried undici.request - the same error - "incorrect header check":

(async () => {
  try {
    const { body, headers, statusCode, trailers } = await client.request({
      path: '/api/ad-server-static_2.xml',
      method: 'GET'
    })
    body.pipe(zlib.createGunzip())
    console.log(`response received ${statusCode}`)
    console.log('headers', headers)
    body.setEncoding('utf8')
    body.on('data', console.log)
    body.on('end', () => {
      console.log('trailers', trailers)
    })

    client.close()
  } catch (error) {
    console.error(error)
  }
})()

[psytechno604]

Ok, in undici/lib/fetch/index.js/httpNetworkFetch/function/onData in the end we have variable "bytes" equal to:

Buffer(809) [31, 139, 8, 0, 0, 0, 0, 0, 0, 255, 237, 148, 223, 111, 218, 48, 16, 199, 223, 251, 87, 120, 121, 79, 2, 180, 116, 180, 10, 169, 24, 180, 19, 210, 218, 85, 37, 116, 147, 170, 42, 114, 99, 15, 172, 134, 56, 179, 205, 175, 253, 245, 187, 24, 2, 9, 93, 43, 173, 15, 123, 152, 78, 66, 2, 238, 206, 119, 223, 59, 159, 63, 193, 197, 106, 150, 146, 5, 87, 90, 200, 172, 235, 52, 189, 134, 67, 120, 150, 72, 38, 178, 73, 215, 25, 71, 87, 110, 199, 185, …]

My attempts always fail because I get from the stream or request something like
Buffer(1438) [31, 239, 191, 189, 8, 0, 0, 0, 0, 0, 0, 239, 191, 189, 239, 191, 189, 239, 191, 189, 111, 239, 191, 189, 48, 16, 239, 191, 189, 239, 191, 189, 239, 191, 189, 87, 120, 121, 79, 2, 239, 191, 189, 116, 239, 191, 189, 10, 239, 191, 189, 24, 239, 191, 189, 19, 239, 191, 189, 239, 191, 189, 85, 37, 116, 239, 191, 189, 239, 191, 189, 42, 114, 99, 15, 239, 191, 189, 239, 191, 189, 56, 239, 191, 189, 205, 175, 239, 191, 189, 239, 191, 189, 239, 191, 189, 24, 2, 9, 93, …]

The reference is the result of zlib.gzipSync, you can see proper headers:
Buffer(950) [31, 139, 8, 0, 0, 0, 0, 0, 0, 19, 237, 150, 91, 111, 218, 48, 20, 199, 223, 247, 41, 178, 188, 39, 1, 90, 58, 90, 5, 42, 6, 237, 84, 169, 237, 170, 2, 221, 164, 170, 66, 105, 226, 129, 213, 16, 103, 182, 185, 237, 211, 207, 73, 19, 110, 193, 52, 147, 120, 216, 195, 223, 66, 74, 176, 79, 206, 61, 39, 63, 247, 114, 49, 9, 141, 25, 225, 130, 178, 168, 105, 86, 237, 138, 105, 144, 200, 103, 1, 141, 70, 77, 115, 208, 191, 182, 26, 230, 101, 235, …]

[psytechno604]

any ideas why undici.stream and undici.request return wrong header?

[mcollina]

I would recommend you to create a small client and server pair to reproduce your problem, it's hard to help you otherwise.

[psytechno604]

(async () => {
  try {
    const { body, headers, statusCode, trailers } = await client.request({
      path: '/api/ad-server-static_2.xml',
      method: 'GET'
    })
    body.pipe(zlib.createGunzip())
    console.log(`response received ${statusCode}`)
    console.log('headers', headers)
    body.setEncoding('utf8')
    body.on('data', console.log)
    body.on('end', () => {
      console.log('trailers', trailers)
    })

    client.close()
  } catch (error) {
    console.error(error)
  }
})()

if I remove the line "body.setEncoding('utf8')" - then the Buffer (that comes into 'data' event) is correct (31, 139, 8, 0, ...) and can be decoded. So, for the "request" method the problem is solved. Please correct the documentation (I took the code from here https://undici.nodejs.org/#/docs/api/Dispatcher?id=example-1-basic-get-request).

I will also check the "stream" method, most likely the root of the issue is the same. But, as soon as there is no explicit encoding setting for body (as for "request"), I am not sure, that it can be fixed without changing the package source code.

[ronag]

Does it work with the normal node http module?

[psytechno604]

of course it works with http module, you can even check the URL in browser - it works and returns always the same static XML: http://ad.service.serverside.ai/api/ad-server-static_2.xml

[dzcpy]

Here is my solution:

      let data: any;
      switch (headers['content-encoding']) {
        case 'br':
          data = await streamToPromise(body.pipe(createBrotliDecompress()));
          break;
        case 'gzip':
        case 'deflate':
          data = await streamToPromise(body.pipe(createUnzip()));
          break;
        default:
          data = await streamToPromise(body);
          break;
      }
      data = /application\/json/.test(headers?.['content-type']) ? JSON.parse(data) : data.toString('utf-8');

[gahabeen]

Using undici.request I had to decompress the body.

Here is my working solution:

import streamToPromise from 'stream-to-promise';
import zlib from 'zlib';

const parseBody = async (response: undici.Dispatcher.ResponseData) => {
    const decoders = []
    const codings = (response.headers['content-encoding'] || '').split(',').map((x) => x.trim())

    for (const coding of codings) {
        if (/(x-)?gzip/.test(coding)) {
            decoders.push(zlib.createGunzip())
        } else if (/(x-)?deflate/.test(coding)) {
            decoders.push(zlib.createInflate())
        } else if (coding === 'br') {
            decoders.push(zlib.createBrotliDecompress())
        } else {
            decoders.length = 0
            break
        }
    }

    return streamToPromise(pipeline(response.body as any, ...decoders, () => { }));
}

Example:

const response = await undici.request(url);
const parsedBody = await parseBody(response);

Hope that helps someone else!