Kannada text issue with ZERO WIDTH JOINER

[MayuraVerma]

There seems to be an issue with "ZERO WIDTH JOINER" and "\u0cb0\u0ccd" combination.
\u0cb0\u0ccd\u200d\u0c95

incorrect display in crome

correct display in ms word

Check the below text in chrome and comparing it with Microsoft word.

ರ ರ್‍ಕ ರ್‍ಖ ರ್‍ಗ ರ್‍ಘ ರ್‍ಙ ರ್‍ಚ ರ್‍ಛ ರ್‍ಜ ರ್‍ಝ

https://en.wikipedia.org/wiki/Kannada_alphabet

[KrasnayaPloshchad]

This is what I have seen on BabelPad

Then I copy the test from here and I produce an image with current version of hb.view.exe
type test.txt | hb-view.exe C:\Windows\Fonts\Nirmala.ttf --output-format=png --output-file=image.png

This is produced with Nirmala UI.

[MayuraVerma]

@KrasnayaPloshchad Thank you for looking into this.

Can u do the same with Noto Sans and Serif Kannada. Is this a problem in harfbuzz or font?

Does babelpad use harfbuzz or directwrite?

[KrasnayaPloshchad]

@MayuraVerma OK I can reproduce them with Noto Sans and Serif Kannada, it seems to me that this is caused by HarfBuzz.

Here is my test with BabelPad

Here is my test with hb-view

BabelPad does not use HarfBuzz and DirectWrite, it relys on Uniscribe and ClearType. Both Chrome and Firefox use HarfBuzz for shaping, also can choose to use DW for glyph rendering.

[MayuraVerma]

@KrasnayaPloshchad Thanks again.

1. Where can I download this utility?
2. You demonstrated Noto Serif Kannada has futher issue with Harfbuzz as reported in Broken Kannada shaping? #347

@behdad Please review this.

[KrasnayaPloshchad]

@MayuraVerma If you want to use this utility, you can go to HarfBuzz website, and then you can find the download link.

[behdad]

I can confirm these. Working on it.

[behdad]

Ok. This is an ugly one :(. @jfkthame I need your help.

The problem is, in most other Indic scripts, a sequence of "ZWJ H C" is used force a subjoined form, whereas in Kannada, the way Tunga font is coded, a "H ZWJ C" is used to force a subjoined form. The problem is, "H ZWJ" is also used to request an explicit half-form. So when we see a sequence "C H ZWJ C", both of these cases can happen, and do happen with Tunga:

• One possibility is that "C H C" would form subjoined, whereas adding ZWJ blocks it and we get half-form followed by base,
• Other case is where "C H C" normally wouldn't form subjoined and ZWJ forces it and we get base followed by half-form.

Since the logic involves complex substitutions in the font including backtrack and lookahead, we cannot know this during initial base selection and have to run lookups to know which happens to be the case. Sure, I can implement extra logic in final reordering to move base forward if we see ZWJ C following base.

However! This is where it gets tricky: some of the right-matras in Kannada should be positioned before-sub. Since we did the sort in initial reordering, they are now between previously decided base, and the later consonants that we want to pick as base. Ouch!

It looks to me that MS Kannada shaper has an extra reordering logic that we have not seen in any other shaper. Here's what I propose: we categorize all right-matras as after-sub. Then in final reordering when adjust base, then go ahead and move those before any below consonants. To do this however, we need a new positional category. I'm not thrilled about that, but let me see what I can do.

[jfkthame]

Hold on.... I'm not sure I understand the issue being reported here, or whether it's being correctly reported.

The reporter is using a sequence RA + Halant + ZWJ + KA (or other trailing consonant).

On Windows 10 (current Insider build), I can reproduce the desired ("correct"?) display (with subjoined KA) as shown in

in BabelPad, but I do NOT see this with MS Word; there, I see the same rendering (with half-forms + full KA) as the reported hb-view result. Maybe my Word is too old (from Office 2010)?

In general, though, I think the correct way to force a subjoined consonant is with ZWJ, Halant, C, and not Halant, ZWJ, C as in the example here. This works for me in all contexts where I've tried it.

I also looked at the behavior of Apple's Kannada font (Kannada MN) on OS X; there, too, the sequence RA + ZWJ + Halant + KA renders as RA + subjoined-KA, while RA + Halant + ZWJ + KA renders as half-RA + KA.

In MS Edge, I do see RA + Halant + ZWJ + KA generating a subjoined KA (i.e. the result is indistinguishable from RA + ZWJ + Halant + KA), but that seems to be specific to RA. I'm guessing it's somehow related to Reph processing, as in the absence of any join-control, the RA + Halant here would form Reph, and the ZWJ suppresses that. But the fact that it then gives a subjoined KA seems to be a non-interoperable quirk, and I wonder if it's really intended/desired, or just an inadvertent quirk of the implementation.

So I'm suggesting that -- perhaps -- this is an incorrect encoded representation of the desired form. They should be using ZWJ + Halant + C for the subjoined consonant. The fact that the behavior of the mis-spelled sequence Halant + ZWJ + C is inconsistent between systems (when it follows RA) is bad, but it's not yet clear to me which result would be correct. (Time to go read TUS, I suppose...)

[roozbehp]

The Unicode section on Kannada seems to confirm @jfkthame's analysis (I didn't go very deep). See http://www.unicode.org/versions/Unicode9.0.0/ch12.pdf, where is shows, in page 498:

U+0CB0 ರ ra + ZWJ + U+0CCD ◌ ್ halant + U+0C95 ಕ ka → ರ‍್ಕ rka

As for how virama+ZWJ should be handled in various scripts, it's a pretty complex story, and I tend to forget the answers. But in short, virama+ZWJ tends to generate half-forms of the previous consonant (or consonant cluster), or in cases where the repha or a repha-like character may not be formed by default (like in Gurmukhi or repaya in Sinhala), form a repha over the second cosonant (or post-ra cluster).

/cc @cibu

[roozbehp]

Of course compatibility with other platforms showing similar rendering is very important to Android, so we'd like this "bug" fixed. But the sequence may not be very frequent (since it can be said it's misencoded), so we can wait until the problem is well understood :)