EUC-jp encoding/decoding support

[r12a]

Results for a series of tests for EUC-jp encoding/decoding can be found at
https://www.w3.org/International/tests/repo/results/encoding-dbl-byte.en#eucjp

The tests can be run from that page (select the link in the left-most column) or get the tests from the WPT repo. There is a PR at
web-platform-tests/wpt#3198

The tests check whether:

1. the browser produces the expected byte sequences for all characters in the euc-jp encoding after 0x9F when encoding bytes for a URL produced by a form, using the encoder steps in the specification.
2. the browser produces percent-escaped character references for a URL produced by a form when encoding miscellaneous characters that are not in the euc-jp encoding. (tests for several ranges)
3. same two types of test when writing characters to an href value
4. the browser decodes all characters as expected from a file generated by encoding all pointers in the euc-jp encoding per the encoder steps in the specification.
5. the browser decodes characters that are not recognised from the euc-jp encoding as replacement characters.

The following summarises the current situation according to my testing, for major desktop browsers. (I will be adding nightly results and perhaps other browsers in time.) The table lists the number of characters that were NOT successfully converted by the test.

Notes:

• Edge fails all href encode tests because characters are not converted to percent-escapes in the href attribute.
• Firefox fails all href encode tests for characters not in the encoding because it converts characters to percent-escaped Unicode values instead.
• eucjp-decode-index: Edge fails on all and only the JIS-X-0212 characters, because it doesn't recognise 0xAF as the first in a 3-byte sequence.

Can we please investigate the failures to ascertain whether:

1. the browser needs to be changed
2. the spec needs to be changed
3. the test is at fault

The following tool may be helpful for investigating issues. It converts between byte sequences and characters for all encodings in the Encoding spec. http://r12a.github.io/apps/encodings/

[r12a]

List of bugs raised:

• https://bugs.webkit.org/show_bug.cgi?id=159889
• https://bugzilla.mozilla.org/show_bug.cgi?id=1285399
• https://bugs.chromium.org/p/chromium/issues/detail?id=626400
• https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8202215/

[jungshik]

As is the case of EUC-KR (form; misc; #62), Chromium's failure in 'form (misc)' in EUC-JP encoding is likely to be caused NOT by Chromium's encoder BUT by Blink's handling of Cf / Default_Ignorable characters.

[hsivonen]

One test showed that the code I had written differed from the spec. However, it seems the test expectation differs from the spec, too:
trail byte outside 0xA1-0xFE: B0 A0 assert_equals: expected "� " but got "�"

Curiously, the spec makes EUC-JP differ from the other two-byte encodings when it comes to the handling of a non-ASCII bogus trail byte. (0xA0 and 0xFF get prepended in addition to ASCII getting prepended.)

Why is EUC-JP different? Is it intentional?

[annevk]

More concretely, you're saying that

If byte is not in the range 0xA1 to 0xFE, inclusive, prepend byte to stream.

should be

If byte is an ASCII byte, prepend byte to stream.

which I think is reasonable and this was probably just an oversight.

Confirmation from @jungshik would be good, but I'm happy to fix this. If I were to fix this, should I make sure I land web-platform-tests at the same time or are you sorting out web-platform-tests @hsivonen?

[hsivonen]

More concretely, you're saying that

If byte is not in the range 0xA1 to 0xFE, inclusive, prepend byte to stream.

should be

If byte is an ASCII byte, prepend byte to stream.

which I think is reasonable and this was probably just an oversight.

I'm saying that that would be consistent with the other two-byte decoders. I haven't investigated legacy decoder behavior on this point, so I can't at this time say whether it should be consistent that way.

If I were to fix this, should I make sure I land web-platform-tests at the same time or are you sorting out web-platform-tests @hsivonen?

I don't have a plan to be sorting out Web Platform Tests. However, I might have to for tests that have already been imported to mozilla-central. (It seems that this one is still in the PR stage on the WPT side.)

[hsivonen]

Firefox (uconv) and Chrome think B0 A0 should result in one REPLACEMENT CHARACTER. Microsoft browsers say two. I'd appreciate it if someone else could test Safari.

[annevk]

Safari Technology Preview yields "X�X".

[annevk]

(Chrome and Safari don't seem to pick a different font by the way, but that's a different class of bugs.)

[hsivonen]

OK, in that case, I think the EUC-JP decoder should prepend ASCII bytes only.

[annevk]

Prepared a PR.

[vyv03354]

Firefox Nightly 56 fixed the encoder, but one decoder test still fails.

[r12a]

Today and yesterday i updated the results at https://www.w3.org/International/tests/repo/results/encoding-dbl-byte.en#eucjp for Firefox, FNightly, Chrome, and Canary. The latest summary is:

[hsivonen]

Firefox Nightly 56 fixed the encoder, but one decoder test still fails.

See discussion upthread. Firefox is correct per spec as amended in May.

[annevk]

Now that Firefox passes all these tests and a year has passed, I'm happy to consider this done. A new issue would also be less noisy at this point, were one warranted.