Regression in Indic shaping of दिसंबर after #3490

[drott]

Bisected to 90f09b1

$ build/util/hb-view ~/dev/blink/src/third_party/test_fonts/test_fonts/Lohit-Devanagari.ttf "दिसंबर"

Expected, at least that's what it was before:

[drott]

This change discussed in #3488 - so perhaps this is the kind of font that is expecting the previous order?

[drott]

    <namerecord nameID="5" platformID="3" platEncID="1" langID="0x409">
      2.95.4
    </namerecord>

lohit_devanagari.zip

That seems to be the "newest" version of the font, according to https://pagure.io/lohit/tree/master

[behdad]

We should test what other shapers do with this. cc @dscorbett

[jfkthame]

Looks like this is a result of GSUB lookup 25, which ligates the i-matra with the anusvara, as part of the pres feature:

      <Lookup index="25">
        <LookupType value="4"/>
        <LookupFlag value="2"/><!-- ignoreBaseGlyphs -->
        <!-- SubTableCount=1 -->
        <LigatureSubst index="0" Format="1">
          <LigatureSet glyph="isigndeva">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvaradeva"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.alt1">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.alt1"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.alt1"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvara.alt1"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.alt2">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.alt2"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.alt2"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvara.alt2"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.alt3">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.alt3"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.alt3"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvaradeva.alt3"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.alt4">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.alt4"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.alt4"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvaradeva.alt4"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.ja">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.ja"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.ja"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvaradeva.ja"/>
          </LigatureSet>
          <LigatureSet glyph="isigndeva.tha">
            <Ligature components="radeva_viramadeva,anusvaradeva" glyph="isign_ra_virama_anusvara.tha"/>
            <Ligature components="radeva_viramadeva" glyph="isign_ra_virama.tha"/>
            <Ligature components="anusvaradeva" glyph="isigndeva_anusvara.tha"/>
          </LigatureSet>
        </LigatureSubst>
      </Lookup>

This lookup ignores base glyphs (because it's expected that the syllable's base is between the i-matra and the anusvara), but since the syllable boundaries are lost, it can now reach across multiple base letters and "grab" an anusvara that doesn't belong here.

I haven't tested what other shapers do with this, but it's clear that this font, at least, is assuming pres operates only within the syllable.

[behdad]

We should test what other shapers do with this. cc @dscorbett

One possibility is that other shapers never clear clusters; that is, they apply all features, including user features, constrained to clusters.

[behdad]

One possibility is that other shapers never clear clusters; that is, they apply all features, including user features, constrained to clusters.

@tiroj have you had evidence that rejects this hypothesis?

[dscorbett]

With HarfBuzz 4.1.0 on Windows:

>hb-shape Lohit-Devanagari.ttf -u 0926,093F,0938,0902,092C,0930
[isigndeva_anusvaradeva=0+266|dadeva=0+541|sadeva=0+709|badeva=4+537|radeva=5+436]

>hb-shape Lohit-Devanagari.ttf -u 0926,093F,0938,0902,092C,0930 --shapers uniscribe
[isigndeva=0+266|dadeva=0+541|sadeva=2+709|anusvaradeva=2@0,-1+0|badeva=4+537|radeva=5+436]

[behdad]

Okay so we regressed.

[jfkthame]

FWIW, the Devanagari spec does say (under Indic input processing),

All shaping operations are done on a syllable-by-syllable basis, independent from other characters

and later under GSUB processing, it says that

the features for presentation forms are applied to the entire cluster simultaneously

which seems to reinforce the view that these features are still being processed on a per-cluster basis.

I think an argument could be made for clearing clusters after all the Indic-specific features have been applied, but before arbitrary user features (though I don't know if other engines do that).

(The other thing maybe worth noting is that according to the spec, GPOS features would also be applied on a per-cluster basis (it says that "[a]ll features are applied simultaneously to the entire cluster"). This would imply that GPOS can't be used to do cross-cluster kerning or other positioning, though, which seems like an unfortunate limitation. So we may want to continue clearing clusters before GPOS, if possible.)

[behdad]

I think an argument could be made for clearing clusters after all the Indic-specific features have been applied, but before arbitrary user features (though I don't know if other engines do that).

The reason we changed this last release is that @tiroj observed that other engines allow user-feature lookups to mix (ie. come before) presentation-form features. That would mean either that other engines don't clear syllables, or implement this in other ways, eg. allowing syllable-sensitivity per lookup.

[behdad]

or implement this in other ways, eg. allowing syllable-sensitivity per lookup.

I've implemented this now.

[behdad]

GPOS features would also be applied on a per-cluster basis

Since we have not been doing that so far, I prefer to keep it that way.

[behdad]

GPOS features would also be applied on a per-cluster basis

Since we have not been doing that so far, I prefer to keep it that way.

In my new implementation though, if pres etc are implemented as GPOS, they will be constrained to the syllable.

[behdad]

While we decide I'm going to revert the regressing commit.

[tiroj]

One possibility is that other shapers never clear clusters; that is, they apply all features, including user features, constrained to clusters.

That is what I documented in 2013: all GSUB in Indic shaping was being applied at the cluster level, which made it impossible to implement cross-cluster contextual GSUB behaviour. See http://tiro.com/John/Problems_for_Indic_Typography.pdf.

At what became the first meeting of the ad hoc OTL working group, in April 2014, we discussed this in detail, and decided that Indic v2 shaping engines should process the rclt feature across cluster boundaries, but to retain the cluster limitation for other GSUB features in case changing the existing behaviour broke existing fonts. The rclt feature was selected for the behaviour change because it was a relatively newly defined feature, so was a safer proposition. See http://tiro.com/John/Fixing_Indic2_OTL.pdf.

What happened after that is that Microsoft and Adobe failed to implement the rclt cross-cluster behaviour in their shaping engines. Apple implemented it in CoreText. HarfBuzz implemented it, and then since then has further relaxed the behaviour to apply additional GSUB across cluster boundaries. It looks like you might have landed on a font that breaks when this change is made, as was raised as a possibility at the 2014 meeting.

[behdad]

HarfBuzz implemented it, and then since then has further relaxed the behaviour to apply additional GSUB across cluster boundaries.

Correct. Previously the way we implemented that disallowed interleaving of lookups of presentation-forms features and user-features. I've now relaxed that limitation.