Make matcher aware of CGJ

[behdad]

Many Arabic fonts have GSUB rules to reorder marks. Currently the matcher skips over CGJ. It shouldn't.

[KrasnayaPloshchad]

So CGJ should be treated the same as other combining characters, isn’t it?

[khaledhosny]

Do we really want to do that? If the font supports <markA><markB> but the user inserts a CGJ between them because normalization re-orders them, skipping CGJ means the font will just work, while not skipping it means the font will have to explicitly support this case or CGJ will be making things worse.

[behdad]

Right now the matches jumps over CGJ because it's a Default_Ignorable. This bug is exactly about not skipping over it, at least when matching marks.

[behdad]

The desired logic here is quite convoluted. We want matcher to NOT skip a CGJ if and only if removing it would have reordered marks...

I think I can do this by changing the CGJ type to a skippable vs non-skippable during normalization. Ie. turn it into a ZWNJ-like thing if it should not be skipped.

[behdad]

Fortunately we have a bit for that already; that's how Mongolian variation selectors are handled. So I just need to turn that bit on for all or certain CGJ's. A good start is to never skip CGJ. I'll put that in now.

[khaledhosny]

With the last commit in master, the third case in w3c/alreq#147 is now broken, but I guess the more refined logic would handle it?

hb-view amiri-regular.ttf --unicodes="0649 0655 034F 0650 062F"

[behdad]

With the last commit in master, the third case in w3c/alreq#147 is now broken, but I guess the more refined logic would handle it?

Yes. I'll work on it. Good to have a test.

BTW, why doesn't Amiri have mark2mark attachments?

[khaledhosny]

BTW, why doesn't Amiri have mark2mark attachments?

It has, but I substitute kasra following hamza with a smaller one and there is no mark to mark attachments between hamza and regular kasra.

[behdad]

I'm releasing 1.6.1. I'm aware that this bug is not fully fixed and 1.6.1 will have different behavior from previous versions, and probably following versions. Just documenting here for posterity.

[KrasnayaPloshchad]

@khaledhosny What happened if you try to workaround for that? For example, add a zero-width glyph and mark attachments for CGJ into Amiri.

[behdad]

We don't want workarounds.