Garbled content when accessing non-utf8 website (Shift_JIS)

[sharmalalit]

I am trying to access a non utf-8 website using request module. Response is garbled for this request.
Even after setting the encoding option to Shift_JIS I am seeing garbled Japanese text.

var request = require('request');
request('http://www.alc.co.jp/', function (error, response, body) {
    if (!error && response.statusCode == 200) {
        console.log(body) // Print the web page.
    }
});

[nylen]

Here are the different encodings, Shift_JIS is not one of them:

http://nodejs.org/api/buffer.html#buffer_buffer

[nylen]

This seems to work:

var concat  = require('concat-stream'),
    Iconv   = require('iconv').Iconv,
    request = require('request');

var conv = new Iconv('Shift_JIS', 'utf8'),
    req  = request('http://www.alc.co.jp/');

req.pipe(conv);

req.on('error', function() {
    console.log('an error occurred');
});

conv.pipe(concat(function(body) {
    console.log(body.toString());
}));

[sharmalalit]

@nylen Thanks a lot for looking into this. Is there a way to detect mid stream that encoding is Shift_JIS and I should use Iconv ?
I can detect using 'Content-Type' header. But some the websites does not define charset in header and does the same using Meta tag in HTML.

Edit : Buffer.isEncoding can serve the purpose here. http://nodejs.org/api/buffer.html#buffer_class_method_buffer_isencoding_encoding

[nylen]

I think that is pretty far outside the scope of the request library. curl and wget don't have this functionality either. The easiest way I can think of is to try phantomjs which is a full-featured headless browser.

Buffer.isEncoding won't help, that just tells whether a given encoding string like utf8 or Shift_JIS is recognized by node.