Error "Invalid UTF-8 sequence" and "Not enough space" when compiling ANSI files (WINDOWS)

[xelloss1012]

Issue

When trying to compile files that are encoding as ANSI (Windows-1252) and have some special characters (i.e. ñ, á, é, etc.), even if this characters are only in a comment, we are getting 2 types of error.

We use this code to compile the files:

sass.render({
        file: filename,
        outputStyle: "nested"            
    }, function (err, result) {
        console.log(err);        
    });

Then, when trying to compile a file with this exact contents:

//á
div {
}

We get this error:

{ Error: Invalid UTF-8 sequence
    at options.error (C:\gitrepos\build-tools\node_modules\node-sass\lib\index.js:291:26)
  status: 1,
  file: 'C:/gitrepos/build-tools/tests/node-sass/v4.8.1/ansi1.scss',
  line: 1,
  column: 3,
  message: 'Invalid UTF-8 sequence',
  formatted: 'Error: Invalid UTF-8 sequence\n        on line 1 of tests/node-sass/v4.8.1/ansi1.scss\n>> //�\r\ndiv {\r\n   --^\n' }

And when trying to compile a file with this other contents:

//aquí todas las reglas que sean necesarias para conseguir la mejor visualización
div {
}

We get a different error:

{ Error: Not enough space
    at options.error (C:\gitrepos\build-tools\node_modules\node-sass\lib\index.js:291:26)
  status: 3,
  message: 'Not enough space',
  formatted: 'Internal Error: Not enough space\n' }

I think both errors are the same, since sometimes changing a little the contents of the file it appears one error or the other.

I have attached the problematic files so you can reproduce the error in an easier way.

test ansi node-sass 4.8.1.zip

Is important to note that this error only happens when the file is encoded as ANSI (Windows-1252). When the file is encoded as UTF-8 it works perfectly. However, even if a file is encoded as UTF-8 (or as ANSI but without any special characters), the error also occurs if any of its import files meets this conditions.

Note that in node-sass 4.7.2 was working great (with libsass 3.5.0.beta.2) so it seems related to the latest release of node-sass 4.8.1 (with libsass 3.5.0).

System info

• NPM version (npm -v): 5.6.0
• Node version (node -v): v8.9.4
• Node Process (node -p process.versions):
  { http_parser: '2.7.0',
  node: '8.9.4',
  v8: '6.1.534.50',
  uv: '1.15.0',
  zlib: '1.2.11',
  ares: '1.10.1-DEV',
  modules: '57',
  nghttp2: '1.25.0',
  openssl: '1.0.2n',
  icu: '59.1',
  unicode: '9.0',
  cldr: '31.0.1',
  tz: '2017b' }
• Node Platform (node -p process.platform): win32
• Node architecture (node -p process.arch): x64
• node-sass version (node -p "require('node-sass').info"):
  node-sass 4.8.1 (Wrapper) [JavaScript]
  libsass 3.5.0 (Sass Compiler) [C/C++]
• npm node-sass versions (npm ls node-sass): node-sass@4.8.1

[xzyfer]

@xelloss1012 thanks for the report. Apologies for the regression. I've confirmed the issues you've mentioned on LibSass 3.5.0.

I'm current debugging #2280 which is relatively urgent. I'll don't expect to get to this before tomorrow.

cc @mgreter you might know where to look off the top of your head.

[mgreter]

This is intended behavior to avoid invalid parsing results (as reported multiple times) and unexpected segfaults when printing error messages etc. We currently only support utf8 (and therefore plain ascii). Supporting single byte encodings should be possible though (detecting charset and replace everything utf8 related with single byte logic) ...

https://www.w3.org/International/questions/qa-choosing-encodings

[xzyfer]

@xelloss1012 apologies I didn't notice the breaking change. As a work around for now you'll need to lock to node-sass@~4.7.

[xelloss1012]

Ok, for me there is no problem, we have version locked to 4.7 to avoid this issue and I think we can live with this by the moment :)

But I suspect this issue can be affecting a number of users, since ANSI is the default encoding used in Windows and in some Windows developer tools like Visual Studio... In special to non english users that may be writing comments with special characters, etc.

Thank you very much for your comments!

[xelloss1012]

Hello again,

We recently have been trying to update to the new node LTS (10.13.0) but then our node-sass that was locked to 4.7 version (to prevent the ANSI problem) does not work, since this version of node-sass is not compatible with node > 9.

As a workaround, we have've been trying to read ourselves the files (with a custom funcion to deal with the ANSI encoding) and then sending the resulting string to the "data" parameter instead of the "file" parameter. This works... unless you have some imported files (mixins, functions, etc.) with the same problem (ANSI files with special characters).

To solve this, we've been trying to use the "importer" parameter. But, since the importer doesn't send you the resolved path but the import string as it is written in the file, in order to use it we need to do some tricky extra work to resolve the filepath... and it seems very dirty and prone to errors (in fact, I still haven't been able to make it to resolve properly some paths)... We currently have something like this:

var customImporter = function (file, prev, done) {
    var getFilePath = function (filePrefix) {
        var p = path.resolve(path.join(process.cwd(), path.dirname(prev), path.dirname(file), (filePrefix || "") + path.basename(file)));
        if (!p.endsWith(".scss")) {
            p += ".scss";
        }
        return p;
    }
    var filePath = getFilePath();
    //If the file does not exist, we try with the prefix
    if (!fs.existsSync(filePath)) { filePath = getFilePath("_"); }
    return { contents: utils.readFileSync(filePath) }
};
sass.render({
    file: file,
    data: utils.readFileSync(file),
    importer: customImporter,
    outFile: outFile
}, function (err, result) {
       ...
});

Before trying to fix my customImporter function, I would like to know if I'm in the right direction or if there is a better way to solve this problem. Maybe some way to just override the "read" function to be used when reading the files to be compiled and their dependencies... or a way to receive a parameter in the importer function with the file path already resolved by libsass, so we can rewrite just the read part without having to reimplement the path resolving...

Thank you in advance.

[xelloss1012]

Just in case someone reads this... :)

I think I've finally come up with the solution. There was some error in the code, so making some slight modifications seems to work properly:

var customImporter = function (file, prev, done) {
    var getFilePath = function (filePrefix) {
        var p = path.resolve(path.dirname(prev), path.dirname(file), (filePrefix || "") + path.basename(file));
        if (!p.endsWith(".scss")) {
            p += ".scss";
        }
        return p;
    }
    var filePath = getFilePath();
    //If the file does not exist, we try with the prefix
    if (!fs.existsSync(filePath)) { filePath = getFilePath("_"); }
    return { contents: utils.readFileSync(filePath), file: filePath }
};
sass.render({
    file: file,
    data: utils.readFileSync(file),
    importer: customImporter,
    outFile: outFile
}, function (err, result) {
       ...
});

[saper]

Not enough space can be explained this way: The byte 0xF3 (ó) indicates that that should be 3-byte UTF-8 sequence, but on that line it is followed only by one byte until CR LF.

[saper]

@xelloss1012 The importer looks good to me, you might only check if file: filePath is good - it could probably be relative name used in @import commands.

But still the best is to convert your source to UTF-8.