Error: unknown compression method

[bitcoinvsalts]

any idea why some url like this one http://www.concordmonitor.com/on-new-hampshire-s-antique-alley-it-s-out-with-the-old-in-with-the-new-9234789 gives this error while calling:

Metascraper.scrapeUrl(results[j].url)

[blakeembrey]

Seems like whatever the page is responding with isn't compressed properly. I'd love to know what exactly it is, other pages with the same headers work using zlib. I can replicate the same behaviour using curl and gunzip on the CLI. E.g.

curl -H 'Accept-Encoding: gzip,deflate' https://github.com | gunzip -dc # works
curl -H 'Accept-Encoding: gzip, deflate' http://www.concordmonitor.com/on-new-hampshire-s-antique-alley-it-s-out-with-the-old-in-with-the-new-9234789 | gunzip -dc # errors with "unknown compression format"

Note: One page is gzip and the other is deflate. Not sure if there's a good page using deflate available to test?

According to https://nodejs.org/api/zlib.html#zlib_compressing_http_requests_and_responses both types of responses should be handled by node.js properly and I haven't seen it before. Anyone else have ideas to fix it? I can try splitting out to the specific methods zlib methods used instead of the combined method. Someone else want to test that fixes it first though? I might not get around to it for a while.

[Kikobeats]

Closing that because 2.0.0 move HTTP calls out of the library, stay stunning!