Fix #36951: Add the "file.encoding_candidate" attribute for selecting encoding to open file.

[a45s67]

TLDR

I try to solve #36951 in the way like "set fileencodings= xxx,xxx" in vim.

---

How to use :

If we want to try to open a file in the order of "utf-8", "big5", "...(some codepages you want)".
In the setting.json, just add -

file.encoding_candidate: [ "utf-8", "big5", "....(some codepages you want)"]

Then the editor will try to open file with "utf-8", if it failed, try "big5", and so on.

Note :

I use "TextDecoder" to check if the file can be opened.
Thus please check that the encoding you want to use exists in "TextDecoder" before use.

Priority:

auto detected encoding < file.encoding_candidate

---

For #36951.

[ghost]

All CLA requirements met.

[bpasero]

Thanks for the PR, a little bit on the expectation: I will probably not have time to review this until we get to our yearly issue grooming iteration in October.

[bpasero]

This seems to be building around the fatal flag on TextDecoder (which I was not aware of). But are we sure this works as we think it does, signalling whether the encoding is valid for a buffer?

https://encoding.spec.whatwg.org/#error-mode

[bpasero]

Besides, we cannot really decode the entire document at once just to check on error for performance/memory reasons and we cannot decode a chunk because we don't really know the byte size...

[bpasero]

Also, TextDecoder with an encoding other than UTF-8 will not work in browsers, so I am inclined to close this PR.

[a45s67]

Thanks @bpasero to give me some comments!

Yes, I did not consider such details you mentioned (fatal flag, byte size) when implementing this feature.
Some code of my commits must make some changes to be more reliable.

To make it work in browsers, maybe I will try to find other text decoder instead of TextDecoder.

But is there a way to check the encoding of a file without decoding all its content? I thought it is necessary to get the accurate result.