Handling of a-gzip compressed content

[nessche]

I am trying to retrieve data using Apple's Appstore Connect API and they return data using a-gzip compression. Zlib is capable of handling it, but since the encoding name is not exactly gzip bent throws an error (which by the way is not caught despite the call being in a try block in my code)

(node:52064) UnhandledPromiseRejectionWarning: TypeError: compression[enc] is not a function

I see two paths to solve this problem, one is adding a key 'agzip' to the compression object defined in nodejs.js and have it pointing to the same function as 'gzip'. Alternatively change the behaviour so that the getResponse method would not try to decompress the content if the encoding/compression is not known (my favourite option).

I can provide a pull-request for either solution, but I would leave the choice of which path to follow to @mikeal

[mikeal]

but since the encoding name is not exactly gzip

What do the transfer encoding headers look like then?

[nessche]

The relevant headers look like these:

Content-Type = application/a-gzip
Transfer-Encoding = chunked
Content-Encoding = agzip

[nessche]

I have a working solution where before piping the response to compression[enc]() I check that such function exists and if not I break out of the while loop surrounding it, thus returning the "raw" buffer which can then be handled by the app-specific code.

[mikeal]

oh wow, this is... odd. Apple is not doing HTTP correctly unless this is double-compressed, which I doubt.

Typically, Content-Encoding is a transport compression and when it’s removed (decompressed) the content should be encoded as whatever Content-Encoding is. In this case, both are a-gzip which is most likely not correct.

As far as a-gzip goes, this is not a commonly used compression identifier and I can’t find much documentation to support it. For example, it’s not noted here at all https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

This is a strange one.

[nessche]

I fed it to the same function that handles gzip and it works just fine, but I also was puzzled to see that it does not match any well known encoding. Which is why, rather than adding agzip to the list of bent known encodings, I'd rather have bent return the raw buffer to the client and let the client deal with it.

FWIW it is not the first time that Apple blatantly misuses standard protocols.... nor will it alas be the last...

[mikeal]

wow, strange. closing for now.

[Gazzini]

For anyone else who comes here -- Apple is, indeed, double-compressing the data.

// let axios decompress it the first time
const response = await axios.get(url, { headers, params, decompress: true, responseType: 'arraybuffer' });
if (response.status === 200) {
        // use gunzip to decompress it again
        const doubleUnzipped = await ungzip(response.data);
        // interpret the data as a string
        const finalText = doubleUnzipped.toString('utf8');
        // done
        console.log(finalText);