Use GBK as fallback, not gb18030

[annevk]

It's equivalent for decoding, but gives more conservative encoding that's likely to be more compatible.

Fixes #4557.

• At least two implementers are interested (and none opposed):
  • Firefox
  • @inexorabletash? See below.
• Tests are written and can be reviewed and commented upon at:
  • This cannot be tested as it depends on the locale of the user agent.
• Implementation bugs are filed:
  • Chrome: N/A, see below
  • Firefox: N/A
  • Safari: https://bugs.webkit.org/show_bug.cgi?id=231660

(See WHATWG Working Mode: Changes for more details.)

---

/parsing.html ( diff )

[inexorabletash]

@JinsukKim ?

[TimothyGu]

I don’t think Chrome puts too much weight on UI language for reasons of predictability. In fact it puts any weight at all only when we have a file scheme URL, from what I can tell. As such, I don’t think this change would have too much of an effect either way as far as Chrome is concerned.

[domenic]

I will approve this as editorially it looks good, but I guess we need to figure out multi-implementer interest. Maybe @rniwa has some thoughts.

[annevk]

@rniwa ping?

@domenic I guess we should also evaluate interest for the status quo as that's clearly not it either, right?

[hsivonen]

FWIW, the only case I'm aware of where Firefox still guesses the encoding from the UI language is the display of non-ASCII file names in FTP directory listings. However, when Firefox guesses something GBK/gb18030-ish from TLD or content, though, it guesses GBK, so I'm in favor of this change.

[domenic]

If we were to merge this change, I'd feel more comfortable if we had a second implementer say something similar to @hsivonen, along the lines of "The current spec doesn't match our implementation, but in spirit this change makes more sense than the current spec". So, @inexorabletash or @JinsukKim, ping again for Blink :).

That said, if this languishes for another week or so without such indications, I'd be OK merging as-is.

I guess we should also evaluate interest for the status quo as that's clearly not it either, right?

Yeah, if we want to sink real time into this, an overhaul of this section or table is needed. Maybe a good next step would be a TLD mapping to replace the UI language mapping.

[hsivonen]

r+ with the comment fixed. Sorry about the delay.

[hsivonen]

Should be:
"windows-936 and GBK are the same"

[annevk]

https://en.wikipedia.org/wiki/Code_page_936_(Microsoft_Windows) does not support this assertion. Do you have a better source?

[annevk]

I made it say "basically the same".

[hsivonen]

Oops, I missed this question. My source is comparing the behavior of GBK in encoding_rs to the code page 936 behavior of the kernel32.dll converter by trying all two-byte combinations. IIRC, the differences relate to ill-formed byte sequences and to Encoding Standard GBK decoder accepting four-byte sequences, and the mapped two-byte sequences matched exactly.

[annevk]

Ah, I guess GBK as per Encoding has 0x80 mapped to the Euro sign, so perhaps it's not quite the same as the "original" GBK. Oh well, I think I improved this comment to a good enough state.

[hsivonen]

Both Firefox and Chrome say GBK here: https://hsivonen.com/test/moz/sniff-zh-hans.htm

[hsivonen]

Thanks for merging.

Both Firefox and Chrome say GBK here: https://hsivonen.com/test/moz/sniff-zh-hans.htm

It turns out that Safari says gb18030 after setting the primary language of the OS to Simplified Chinese and rebooting. (Note: Setting Safari's language to Simplified Chinese and relaunching Safari is not enough.)

[annevk]

I filed https://bugs.webkit.org/show_bug.cgi?id=231660 against WebKit.