fs: use WTF-8 on Windows

[StefanKarpinski]

I got tired of waiting on #2192, so I just applied the code review suggestions made by @vtjnash. Hopefully I'm doing the return error codes right, but if not, let me know what else to do here.

There were a couple of formatting recommendations for long function signatures that I did not follow since lining wrapped up arguments with the opening parens does not seem to be the existing style in this file. Instead, I indented the wrapped arguments to be indented once after the normal function body level, which seems to be how the rest of this file is formatted. If the other format is preferable, it would be better to make a separate cosmetic PR to fix that.

[StefanKarpinski]

Update: this now threads ERROR_INVALID_NAME back to the caller if any path name is invalid WTF-8. I also made the types used a bit more specific: code points are always int32_t, path lengths are always int (plenty for paths on all platforms); the types are signed because -1 is used to signal invalid WTF-8 data. I put some asserts in there that are strictly not necessary since they're after functions that cannot return invalid values (i.e. going from WTF-16 to WTF-8), but it seemed better to put the asserts in anyway with a comment.

[StefanKarpinski]

Ok, I applied your suggestions but I ended up rewriting fs__decode_wtf8_char entirely to be simpler.

[seishun]

The function fs__decode_wtf8_char includes validation (albeit incomplete), which was deliberately removed from the spec (see SimonSapin/wtf-8@c6a271d). This implies one of the following:

1. The spec's author was wrong in limiting the algorithm to well-formed WTF-8
2. libuv is a special case

Which one is it?

[StefanKarpinski]

The code raises an error if the WTF-8 is not well-formed, which is exactly what the spec suggests.

[seishun]

Where does it suggest that?

[StefanKarpinski]

The spec doesn't actually address ill-formed WTF-8 at all as far as I can tell, but it does suggest when converting strictly from WTF-8 to UTF-8 that one return an error if the WTF-8 is not valid UTF-8. The other option that it suggests is replacing unparied surrogates with replacement characters. The situation going from arbitrary bytes to WTF-8 is analogous and the viable choices are the same: raise an error or replace the ill-formed sequences with replacement characters. Since the spec explicitly doesn't address anything besides well-formed WTF-8, we're free to do which of these we feel is appropriate. The most conservative thing to do is to raise an error with the caller, which is precisely what this pull-request does.

[seishun]

The spec doesn't actually address ill-formed WTF-8 at all as far as I can tell

Correct, and the note in http://simonsapin.github.io/wtf-8/#decode-from-wtf-8 explains why.

but it does suggest when converting strictly from WTF-8 to UTF-8 that one return an error if the WTF-8 is not valid UTF-8

From well-formed WTF-8. This is a different situation.

Since the spec explicitly doesn't address anything besides well-formed WTF-8, we're free to do which of these we feel is appropriate.

It doesn't address ill-formed WTF-8 because it assumes that this situation will never occur. One could see it as a litmus test for suitability of WTF-8 for a given purpose. The fact that you seem to argue that this situation can occur suggests that WTF-8 is not being used for its intended purpose.

[StefanKarpinski]

You can view this PR as doing two separate things:

1. Verify that caller-supplied data is well-formed WTF-8; if it isn't raise an error to the caller.
2. Given well-formed WTF-8, convert it to a sequence of code points as per the WTF-8 spec.

The spec only applies to the second step, and that step is done precisely according to spec. The first step is, by the spec's own statement, outside of scope of the spec. The spec explicitly says that it has no position about how this step should be handled. That doesn't mean that we can't apply the WTF-8 spec to the second step anymore than the fact that the Unicode spec doesn't discuss gzip means that we can't decompress a gzipped file and then interpret the decompressed data as UTF-8. If the gzip file is malformed, then we throw an error before trying to decode the UTF-8 data. Your argument is like saying that because gzip files can be incorrectly formatted, we can't possibly apply the Unicode standard to interpret the contents of a decompressed gzip file.

Even though the WTF-8 spec doesn't address ill-formed WTF-8 data, we can extrapolate from the options it gives for handling converting WTF-8 to UTF-8. There are two approaches:

1. The strict, non-lossy approach where you throw an error upon encountering any unpaired surrogate;
2. The lossy, non-strict approach where you convert unpaired surrogate code points to replacement characters.

The same options exist for handling step one above:

1. The strict, non-lossy approach where you throw an error upon encountering any ill-formed WTF-8;
2. The lossy, non-strict approach where you convert ill-formed WTF-8 sequences to replacement characters.

Again: the spec doesn't address this explicitly, but the fact that step one is out of scope for the WTF-8 spec doesn't mean that we can't apply the WTF-8 spec to step two. In this PR, we take the more conservative approach here and raise an error to the caller when they pass ill-formed WTF-8.

[StefanKarpinski]

It doesn't address ill-formed WTF-8 because it assumes that this situation will never occur.

Btw, this is your interpretation, not something the spec says. I suspect the spec authors are aware that ill-formed WTF-8 can and does occur, they just consider it out of scope of the WTF-8 spec to tell you how to handle it. That doesn't mean that the problem goes away, or that if you have a situation where it can occur you just stick your head in the sand and ignore the possiblity. It just means that it's up to you to decide how to handle it.

[seishun]

I don't know how you ended up with that interpretation if you've read the note I linked. The analogy is flawed because validation is an integral part of any encoding spec where potentially ill-formed input can occur (e.g. https://encoding.spec.whatwg.org/#utf-8-decoder) - and WTF-8 isn't one.

ill-formed WTF-8 can and does occur

How can it occur in a self-contained system?

[vtjnash]

This is ready to merge

[StefanKarpinski]

w00t!