Khmer shaping does not match Uniscribe or CoreText

[punchcutter]

Brought up by @mcdurdin at https://twitter.com/MarcDurdin/status/941516844195749888
Text is
ស៊ើប

1. ស៉ើុប
2. ស៉េីុប
3. ស៉ីេុប
4. ស៉ើុុប
5. ស៉េីុុប
6. ស៉ីេុុប
7. ស៊ើុប
8. ស៊េីុប
9. ស៊ីេុប
10. ស៊ើុុប
11. ស៊េីុុប
12. ស៊ីេុុប
13. ស៉ើីប
14. ស៉ីើប
15. ស៉ើីុប
16. ស៉ីើុប
17. ស៊ើីប
18. ស៊ីើប
19. ស៊ើីុប
20. ស៊ីើុប
21. សើីុប
22. សើុីប
23. សីើុប
24. សីុើប

Windows 10 Edge and OS X 10.12.6 TextEdit agree that only one combination can be rendered correctly:

harfbuzz (here Firefox Nightly, but also tested with latest hb-view) does not agree and allows all of these combinations to render.

Font handling of consonant shifters can be different depending on the font which is why the Android test with an older version of Noto Khmer showed everything rendering exactly the same. The new version of Noto Khmer doesn't allow that to happen (only 7-12 above look correct even though they aren't), but either way the shaper shouldn't allow every one of these to shape the same.

[mcdurdin]

@MakaraSok is currently working on providing additional data for this issue. Due to some ambiguity in the Unicode and OpenType specifications for Khmer, it is not immediately clear which combinations should be permitted and which should be blocked, especially in the boundary between syllables. Also important to consider how minority languages use the Khmer script to avoid blocking them in any fixes.

There are also 12 more basic examples for this syllable (screenshot from presentation):

Copying @jahorton, @mhosken

[behdad]

The same happens to all Indic scripts as well I believe. We don't enforce the one-matra per-position rule, neither do we enforce a specific matra order. cc @jfkthame

[jfkthame]

A strict one-matra-per-position rule would be problematic, because of things like हााााााा (yes, real-world examples of that kind of thing do happen, e.g. in comic-book text).

[mcdurdin]

@jfkthame Yes, I don't see that as an issue given that is visually different. The issue described here relates only to multiple visually-indistinguishable sequences.

[behdad]

Jonathan and I separated the Khmer shaper from the Indic one. Next step is to remove more code from the Khmer shaper, then code in the matra ordering into the grammar.

[Richard57]

The Khmer encoding was designed to have only one matra per syllable, despite have commonality with Thai writing habits in that respect. There should be a looming issue with the Khom script (i.e. the variety of the Khmer script used in Thailand for Pali and some writing in the Tai vernaculars), as all three of the apparent signs combinations <E, [I, II, Y]> have been in use at some time since the middle of the 19th century, and it is likely that they have been used for a 2-way length contrast, as in the Lao Lao script writing system. There has also been variation in Khmer, though the temptation is to dismiss that as glyph variation.

[behdad]

Can someone test this again? Our Khmer shaper is much closer to spec / uniscribe now.

[MakaraSok]

@behdad Thank you for your work on this. I'm a Khmer native speaker and I'm willing to help testing Khmer script, please give me some guidance on how to test it.