Khmer consonant shifter behavior with ZWNJ

[iongchun]

Hi, I am testing rendering of Khmer words which need ZWNJ.
One example is U+179a 1784 17d2 179f 200c 17ca 17b8.

Testing with KhmerOS Regular and Uniscribe, the word without ZWNJ (U+179a 1784 17d2 179f 17ca 17b8) is shaped as:
[uni179a=0+775|uni1784=1+1550|uni179f.sub=1+775|uni17bb=1@-125,-700+0|uni17b8=1+0]
and with ZWNJ, uni17ca is not substituted with uni17bb:
[uni179a=0+775|uni1784=1+1550|uni179f.sub=1+775|space=1+0|uni17ca=1+0|uni17b8=1@0,300+0]

Now with harfbuzz indic shaper, the word without ZWNJ:
[uni179a=0+775|uni1784=1+1550|uni179f.sub=1+775|uni17bb=1@-125,-700+0|uni17b8=1+0]
with ZWNJ, uni17ca is still substituted with uni17bb:
[uni179a=0+775|uni1784=1+1550|uni179f.sub=1+775|space=1+0|uni17bb=1@-125,-700+0|uni17b8=1+0]

From Microsoft's document, substitution of RegShift should be implemented with 'blwf' feature in fonts.
It seems true for KhmerOS and Microsoft's Khmer UI: if I mask the BLWF flag for the consonant shifter after ZWNJ, the word is shaped correctly.

But with Mondulkiri it is not true, BLWF has no affect. The word without ZWNJ is shaped as:
[uni179A=0+715|uni1784=1+1465|uni17D2179F=1+675|uni17BB.low=1@0,100+0|uni17B8=1+0]
and with ZWNJ:
[uni179A=0+715|uni1784=1+1465|uni17D2179F=1+675|space=1+0|uni17BB.low=1@0,100+0|uni17B8=1+0]
For reference the word with ZWNJ is shaped by Uniscribe as:
[uni179A=0+715|uni1784=1+1465|uni17D2179F=1+675|space=1+0|uni17CA.lower=1+0|uni17B8.ms.high=1+0]

I guess for Mondulkiri, if substitution (abvs in this case) lookups do not ignore ZWNJ (as ZWJ), it might yield better result.
Then after some experiment, viola! Mondulkiri looks good with harfbuzz as with Uniscribe!

Since I am a font newbie, these analysis and fix might make no sence ;)
I have the code in my branch, and got some result from the tests.

[KrasnayaPloshchad]

I can also produce this on Windows 10 with Leelawadee UI font.

First I get the text to reproduce on BabelPad, which using Uniscribe:

The following screenshot is what I have seen when I turn off the complex layout (click “U˙˙” button), the “pipe” mark is ZWNJ

Then I reproduce via hb-view
type input.txt | hb-view.exe C:\WINDOWS\Fonts\LeelawUI.ttf --output-format=png --output-file=image.png --utf8-clusters

LeelawUI.zip

[iongchun]

Thanks @KrasnayaPloshchad
Both Leelawadee UI and Khmer UI give wrong position to U+17B8, with Uniscribe or not.
So I guess it is a font issue.

[KrasnayaPloshchad]

@iongchun OK you’re right that Leelawadee UI really has wrong position to U+17B8, it seems to me that ZWNJ was ignored when produce this word with HarfBuzz.

[jungshik]

It'd be interesting to see how Noto Sans Khmer, Noto Sans Khmer UI and Noto Serif Khmer behave.

[iongchun]

I tested Noto Sans Khmer, my patch does improve in the word example with ZWNJ: U+179a 1784 17d2 179f 200c 17ca 17b8
Shaped by harfbuzz:
[uni179A=0+588|uni1784=1+1300|uni17D2179F=1+588|space=1+0|uni17BB.b=1+0|uni17B8=1+0]
Shaped by Uniscribe, and harfbuzz with this patch:
[uni179A=0+588|uni1784=1+1300|uni17D2179F=1+588|space=1+0|uni17CA=1+0|uni17B8.a=1+0]

But it has another problem: for the combination of SA (U+179F) and coeng RO (U+179A), it does not change the form of the consonant shifter:
U+179f 17ca 17d2 179a 17b9 1794 (SA shifter coeng RO Y BA; Unicode 4+)
Uniscribe: [uni17D2179A=0+588|uni179F=0+1900|uni17BB=0+0|uni17B9=0+0|uni1794=5+1300]
harfbuzz: [uni17D2179A=0+588|uni179F=0+1900|uni17CA=0+0|uni17B9.a=0+0|uni1794=5+1300]

U+179f 17d2 179a 17ca 17b9 1794 (SA coeng RO shifter Y BA; Unicode 3)
Uniscribe: [uni17D2179A=0+588|uni179F=0+1900|uni17BB=0+0|uni17B9=0+0|uni1794=5+1300]
harfbuzz: [uni17D2179A=0+588|uni179F=0+1900|uni17CA=0+0|uni17B9.a=0+0|uni1794=5+1300]

Noto Serif Khmer is another story. It does not shape the word example with ZWNJ correctly even with this patch.
And it still change the form of consonant shifter for the combination of SA (U+179F), coeng RO (U+179A), and ZWNJ:
U+179f 17d2 179a 200c 17ca 17b9 1794 (SA coeng RO ZWNJ shifter Y BA; Unicode 3)
Uniscribe:
[uni17D2179A=0+713|uni179F=0+2162|space=0+0|uni17CA=0+0|uni17B9=0+0|uni1794=6+1475]
harfbuzz:
[uni17D2179A=0+713|uni179F=0+2162|space=0+0|uni17BB=0+0|uni17B9=0+0|uni1794=6+1475]

U+179f 200c 17ca 17d2 179a 17b9 1794 (SA ZWNJ shifter coeng RO Y BA; Unicode 4+)
Uniscribe:
[uni17D2179A=0+713|uni179F=0+2162|space=0+0|uni17CA=0+0|uni17B9=0+0|uni1794=6+1475]
harfbuzz:
[uni17D2179A=0+713|uni179F=0+2162|space=0+0|uni17BB=0+0|uni17B9=0+0|uni1794=6+1475]

I never tested with Noto Sans Khmer UI though.

[punchcutter]

I think there are two issues here:

1. Font issues - the fonts mentioned don't all have the same lookups even though they really should, especially for register shifters.
2. harfbuzz does not behave like Uniscribe (correct or not is another question, but from the above tests it's clear that the results are different)

You shouldn't waste any time looking at Noto Serif Khmer or Noto Sans Khmer UI because they are broken in many ways. The next version will have the same lookups as the latest Noto Sans Khmer. Noto Khmer handles register shifters pretty much the same as Mondulkiri in the blwf feature, but Mondulkiri duplicates some of the lookups in the abvs. I'm not sure if that was a deliberate decision, but it's rather odd and not really the correct way to do it. In the above sequence U+179f 17ca 17d2 179a 17b9 1794 Mondulkiri displays correctly in harfbuzz only because the lookup is in the abvs. Somehow harfbuzz is skipping the blwf lookup. Since Uniscribe displays these sequences correctly with and without ZWNJ (Mondulkiri and Noto Khmer) it seems to be a harfbuzz issue. If I apply the above changes from @iongchun to harfbuzz then both Mondulkiri and Noto Khmer display correctly for me.

[KrasnayaPloshchad]

cc @khaledhosny

[KrasnayaPloshchad]

I think this is not a simply issue between font and OT backend, this is also a issue for which method is respect their orthography.
@punchcutter If you are really not sure which is correct, you can ask Cambodians for help.

[KrasnayaPloshchad]

@iongchun Do you want to submit a pull request?

[KrasnayaPloshchad]

CC @jfkthame

[iongchun]

Pull request submitted: #353

[KrasnayaPloshchad]

Here is my test results with other two fonts included in Windows:

• DaunPenh
  
  
• MoolBoran
  
  

Both of them works proper on Uniscribe based environment, and give right position to U+17B8, but their ZWNJ does not works with them on HarfBuzz based environment.