Wrong guess encoding as Windows 1252

[maooyer]

Upstream issue: aadsm/jschardet#48

• VSCode Version: Version 1.15.1
• OS Version: Windows 10.0.15063

Steps to Reproduce:

1. The settings.json of my vscode

    "files.encoding": "utf8",
    "files.autoGuessEncoding": true,

2. Ceate two txt files, make sure the files are saved as utf-8

test1.txt

Created on: 2017年9月2日
测

test2.txt

Created on: 2017年9月2日
测试

3. Reopen the files,test1.txt guessed encoding is Windows 1252 and test2.txt guessed encoding is utf-8.

Reproduces without extensions: Yes

[rodolfomuller]

I have same problem with utf8 and iso88591.

[sou-lab]

I had same problem.
However, it was maybe fixed in 1.17.

Please try Insiders.
https://code.visualstudio.com/insiders/

[codecrafting-io]

Well I tried on 1.17 and still happens. On my case a blank txt file with some word with accents even saving with UTF-8 still reopens with Western 1252 or ISO 8859-2.

[sou-lab]

Sorry. not fixed by 1.17.0.
I still happens too😢
but, Insiders guessed encoding is UTF-8.

[Yanpas]

Still not fixed in 1.23:
Test case

#!/bin/sh

foo() {
	echo "starting …"
}

Ellipsis symbol … makes vscode guess cp1252

[std4453]

Any updates? I'm still getting this issue today.

[Yanpas]

Could the fix in this issue #23997 lead to regression?

[Yanpas]

The code from latin1prober.js:113 looks very suspicious

    this.getCharsetName = function() {
        return "windows-1252";
    }

It's the cause of the problem.

For those of you who are interested in debugging here is a snippet:

const detect = require("./init").detect
const fs = require('fs')
let args = []
process.argv.forEach(v => args.push(v))
let fname = args[2]
let buf = fs.readFileSync(fname)

console.log(detect(buf))

Place it to src/tst.js. To run call node tst.js somefile from src folder

[chylex]

Same issue here... autoGuessEncoding is disabled and yet VS Code still attempts to guess encoding as Windows-1252 even though that causes invalid characters because it's actually UTF-8...

[elmonty]

It happens to me when there is a copyright symbol in the file. VSCode incorrectly guesses Windows-1252, which shows an invalid character next to the copyright symbol.

[MxDany]

omg, Hard to believe that this problem actually existed for so long, 1.40.2 Still not fixed...

[JulioNobre]

How to upvote?

[codecrafting-io]

@JulioNobre I get that may be ambiguous to identify the exact encoding, but something has to be done, even that does not involve a Microsoft code directly. Try to create a blank txt file with the Windows-1252 encoding and write the word "coração". Now open the file, and you still see that even something aparently simple and created by Code, the guessed encoding still wrong. I tried to simulate this on multiple Text Editors, and no one opened the file with wrong encoding. I also testd with ISO 8859-1, same issue. So this is a problem, because if the application offers the options to save with multiple encoding, it should at least open the file created with the same encoding, otherwise don't offer certain encoding options.

[cv0cv0]

VSCode 1.41.0 Still not fixed...

[deadbaed]

still not fixed, version 1.42.1

[deadbaed]

although i got it working with a workaround:

in my settings.json file i disabled guessing the file encoding and i force files to be encoded in utf-8:

    "files.encoding": "utf8",
    "files.autoGuessEncoding": false

[ghost]

although i got it working with a workaround:

in my settings.json file i disabled guessing the file encoding and i force files to be encoded in utf-8:

    "files.encoding": "utf8",
    "files.autoGuessEncoding": false

"files.autoGuessEncoding" can also be disabled per language, which mitigates the problem. It would be helpful to be able to do per file extension.

[savioret]

although i got it working with a workaround:

in my settings.json file i disabled guessing the file encoding and i force files to be encoded in utf-8:

This only has sense if your whole project is encoded as UTF-8
autoGuessEncoding is precisely useful for projects having mixed encoded files

[std4453]

Still experiencing this problem.
Forced to reopen with UTF-8 every time I open the file.
At least it should try to save my choices.
Annoyed.

[haba713]

People, on the 29th of May we have had this issue open for 1000 days.

Celebrate... 🍰

[hmcoder-zz]

1002 today..

[lingsamuel]

This issue originally describes this bug: aadsm/jschardet#56, which should be fixed by aadsm/jschardet#57 and aadsm/jschardet#59.

[lingsamuel]

I think this newest release solves this issue: https://github.com/aadsm/jschardet/releases/tag/v2.2.1

@bpasero

[bpasero]

We can pick up a new version for July, as we are currently closing for June endgame.