USE shaping with ZWJ/ZWNJ

[punchcutter]

The USE spec isn't entirely clear on how ZWJ and ZWNJ behave as part of a cluster since they aren't included in the cluster validation models. Including them in a sequence can have unexpected results.

The situation that revealed this was in the Buginese ligature 1A15 1A17 200D 1A10 as mentioned in the Unicode Standard. The USE shaper first splits the sequence into USE clusters which don't include ZWJ so this becomes three clusters instead of one. Since ccmp is one of the features applied per cluster rather than per run the ligature never forms. Move it to a liga feature and it's fine because that is applied per run.

If ZWJ/ZWNJ are intended to be processed as part of a valid USE cluster then something needs to change in harfbuzz as well as every other shaper I know of. It seems to be a rare situation, but here we had it and looking at some Devanagari I also see half forms being done by adding ZWJ to the end of a sequence. Currently that gets run through the Indic shaper so no problem, but if any scripts get run through USE and try to use ZWJ or ZWNJ in any of the per-cluster features (locl, ccmp, nukt, akhn, rphf, pref, rkrf, abvf, blwf, half, pstf, vatu, cjct) then things might not work as expected.

[Richard57]

ZWNJ also induces dotted circles if inserted before matras. This is a problem if one wants to suppress the ligature of U+1A45 NA and U+1A63 SIGN AA in Tai Tham, as in the word ᨴᩤᩴᨶ᩠ᩅ‌ᩣ᩠ᨿ /tamnwaːi/ 'to foretell' on p285 of the Northern Thai Dictionary of Palm-Leaf Manuscripts, or in didactically displaying the unligated cluster. The USE only mandates two dotted circles (and only one in Unicode 10.0) in the word 'to foretell' written without the ligature, but both the USE (at least, in May 2017) and HarfBuzz 1.5.1 produce three. Both renderers are in breach of the USE specification. The word is encoded 1A34 1A64 1A74 1A36 1A60 1A45 200C 1A63 1A60 1A3F. (By contrast, HarfBuzz allows U+200C between U+0BA3 TAMIL LETTER NNA and U+0BBE TAMIL VOWEL SIGN AA.) The Maefahluang dictionary gives only the by-form ᨲ᩵ᩣᩴᨶ᩠ᩅᩣ᩠ᨿ, which has a different tone on the first syllable, which it shows with the ligature for the second syllable.

[Richard57]

I forgot I'd also commented on this issue at #483 .

[punchcutter]

This is also a problem with Newa. In TUS it says half forms can be produced with ZWJ after Virama just like Devanagari, but since Newa is run through the USE shaper the ZWJ isn't processed as part of the cluster. This will also be a problem with Devanagari if existing fonts are updated to use dev3 (or other Indic v3 (#539) and expect to do half forms the same way. I tested by adding ZWJ to the standard_cluster like

consonant_modifiers = CMAbv* CMBlw* ((H ZWJ? B | SUB) VS? CMAbv? CMBlw*)*;

and everything seems to work as expected, but I don't know if that's the best way to handle it here.

@behdad What do you think? We kind of need this to work to be able to test and deliver Noto Newa.

[behdad]

cc @jfkthame
I should bring this up with Andrew Glass and see what he has in mind. Jonathan and I tried before changing the grammar to allow joiners everywhere but we couldn't finish that work the way we wanted. Just adding them in places we suspect are useful is another alternative; I'm open to that.

[punchcutter]

@behdad Any word from Andrew Glass about this?

[behdad]

I dropped the ball. Let me ping him quickly.

[behdad]

Andrew says:

USE defines ZWJ and ZWNJ as follows:

Zero-width non-joiner (ZWNJ)

The zero-width non-joiner is used to prevent a fusion of two characters. It continues a preceding cluster but causes a cluster break after itself when the following character is not a mark character (gc=Mn or gc=Mc). ZWNJ does not reset the cluster model.

Zero-width joiner (ZWJ)

The zero-width joiner is used to fuse two characters. It continues a preceding cluster and joins it to a following character unless the following character is another ZWJ. In which case there will be a cluster break between the two ZWJs. ZWJ does not reset the cluster model.

These aren’t enumerated in the cluster model because it would be too noisy to read.

So ZWJ is allowed anywhere inside a cluster. And ZWNJ is allowed inside a cluster when the following character is either Mn or Mc. Otherwise ZWNJ is the final character in the cluster. For example | shows cluster boundaries in the following:

| Lo ZWJ Lo |
| Lo ZWJ Mn |
| Lo ZWNJ | Lo |
| Lo ZWNJ Mn |

I agree this isn’t working for Buginese, so that is a bug in Windows’ implementation.

[Richard57]

There are very few cases where one could actually have "| Lo ZWJ Lo |". The first letter would have to be Consonant_With_Stacker, Consonant_Prefixed or Consonant_Preceding_Repha.

[punchcutter]

Looking at all mentions of ZWJ in UTS it seems to me like the most common places they might appear in USE shaping are either before a virama or after it. This should cover the usual half and rphf uses of ZWJ. Even something like the Devanagari below base rakaar form happening in blwf should work here (00A0 200D 094D 0930). I'm sure something else could come up, but I propose we at least allow ZWJ before and after virama. Something like

consonant_modifiers = CMAbv* CMBlw* ((ZWJ? H ZWJ? B | SUB) VS? CMAbv? CMBlw*)*;

I believe that will cover the majority of cases where current USE shaping will fail.

[Richard57]

Punchcutter's suggestion would only help with scripts that suffer from virama. The problems reported are for Buginese, which has nothing matching 'H', and Tai Tham, which has an invisible stacker, i.e. a coeng. Buginese is to use ZWJ where other scripts use an invisible stacker, and Tai Tham has optionally ligating marks. The only thing that comes near in Tai Tham is the HarfBuzz-defeating, unattested possibility of *ᨬᩮ‌᩠ᨬ /ɲeːn/ 1A2C 1A6E 200C 1A60 1A2C as distinct from ᨬ᩠ᨬᩮ /-n ɲeː/ 1A2C 1A60 1A2C 1A6Ewithout ZWNJ. Tai Tham has a coeng, not a virama. (Read /j/ for /ɲ/ for the northern languages.) As USE allows substitutions before Indic rearrangement, it is possible to write *ᨬᩮ᩠ᨬ without ZWNJ, but this might not work with other engines. (I'm currently only exploiting this capability with my font that distinguishes subscript consonants in the coda from those in the offset - Da Lekh Si - the distinction might provide better support for spell-checkers.)

[punchcutter]

I only mentioned Buginese originally as a very simple example of where it's easy to see the difference between features applied per validated cluster (ccmp) and features applied to the whole run (liga). Either way Buginese is fine without any shaper change. There's only one ligature formed with ZWJ and that works fine when it's under the liga feature. My proposal right now is to start adding support where we absolutely know it will be needed and as I wrote above we absolutely need it for Newa. Obviously this won't cover every possible situation that might come up with every script, but those can be addressed later. If we want Newa to work on Android when we deliver it then we need to fix this now.

[Richard57]

Does rendering Newa have problems with 'top' and 'right' matras that actually behave as though they had a left matra component with some consonants? (I have almost no knowledge beyond https://www.unicode.org/L2/L2012/12003r-newar.pdf Section 3.6.) Or are you resigned to the font doing Indic rearrangement, as I have to do when repairing the damage done by USE to Tai Tham?

[punchcutter]

@Richard57 No problems with the Newa for that. Those vowels get split into two parts in the pref feature (before USE reordering) and then the pre-base is automatically reordered. The only thing not working for Newa right now is half forms and repha because those require ZWJ.

[behdad]

Thanks Zachary. I did just as you requested. Keeping this issue open to address the rest of what Andrew specified.