Chakma vowel decomposition (1112E and 1112F) not working with USE

[punchcutter]

These vowel signs are getting a dotted circle inserted between them instead of just decomposing. Looking at the shaping code they are decomposed in hb-ot-shape-complex-indic.cc, but Chakma is shaped with USE so shouldn't the decomposition be in hb-ot-shape-complex-use.cc? There are also other decompositions in the indic shaper for scripts shaped by USE:

    case 0x0F77u  : *a = 0x0FB2u; *b= 0x0F81u; return true;  // Tibetan
    case 0x0F79u  : *a = 0x0FB3u; *b= 0x0F81u; return true;  // Tibetan
    case 0x17BEu  : *a = 0x17C1u; *b= 0x17BEu; return true;
    case 0x17BFu  : *a = 0x17C1u; *b= 0x17BFu; return true;
    case 0x17C0u  : *a = 0x17C1u; *b= 0x17C0u; return true;
    case 0x17C4u  : *a = 0x17C1u; *b= 0x17C4u; return true;
    case 0x17C5u  : *a = 0x17C1u; *b= 0x17C5u; return true;
    case 0x1925u  : *a = 0x1920u; *b= 0x1923u; return true;  // Limbu
    case 0x1926u  : *a = 0x1920u; *b= 0x1924u; return true;  // Limbu
    case 0x1B3Cu  : *a = 0x1B42u; *b= 0x1B3Cu; return true;  // Balinese
    case 0x1112Eu  : *a = 0x11127u; *b= 0x11131u; return true;  // Chakma
    case 0x1112Fu  : *a = 0x11127u; *b= 0x11132u; return true;  // Chakma
    case 0x0B57u  : *a = no decomp, -> RIGHT; return true;
    case 0x1C29u  : *a = no decomp, -> LEFT; return true;  // Lepcha
    case 0xA9C0u  : *a = no decomp, -> RIGHT; return true;  // Javanese
    case 0x111BuF  : *a = no decomp, -> ABOVE; return true;

Shouldn't these all be in hb-ot-shape-complex-use.cc? To test Chakma I added a decompose function to hb-ot-shape-complex-use.cc and it works as expected:

static bool
decompose_use (const hb_ot_shape_normalize_context_t *c,
		 hb_codepoint_t  ab,
		 hb_codepoint_t *a,
		 hb_codepoint_t *b)
{
  switch (ab)
  {
    case 0x1112Eu  : *a = 0x11127u; *b= 0x11131u; return true;
    case 0x1112Fu  : *a = 0x11127u; *b= 0x11132u; return true;
  }


  return (bool) c->unicode->decompose (ab, a, b);
}

static bool
compose_use (const hb_ot_shape_normalize_context_t *c,
	     hb_codepoint_t  a,
	     hb_codepoint_t  b,
	     hb_codepoint_t *ab)
{
  /* Avoid recomposing split matras. */
  if (HB_UNICODE_GENERAL_CATEGORY_IS_MARK (c->unicode->general_category (a)))
    return false;

  return (bool)c->unicode->compose (a, b, ab);
}


const hb_ot_complex_shaper_t _hb_ot_complex_shaper_use =
{
  "use",
  collect_features_use,
  NULL, /* override_features */
  data_create_use,
  data_destroy_use,
  NULL, /* preprocess_text */
  NULL, /* postprocess_glyphs */
  HB_OT_SHAPE_NORMALIZATION_MODE_COMPOSED_DIACRITICS_NO_SHORT_CIRCUIT,
  decompose_use,
  compose_use,
  setup_masks_use,
  HB_OT_SHAPE_ZERO_WIDTH_MARKS_BY_GDEF_EARLY,
  false, /* fallback_position */
};

A simple test: 𑄇𑄮
Before the change 11127 gets a dotted circle. After the change it decomposes properly. This font has other issues, but for this test it seems to work.
SuzMoriz.ttf.zip

[KrasnayaPloshchad]

What happened if you try it with BabelPad?

[punchcutter]

@KrasnayaPloshchad BabelPad and Edge in Windows 10 seem to be ok. WebKit nightly in macOS Sierra doesn't work correctly. It just uses the encoded glyph (1112E or 1112F). Harfbuzz is breaking it into the components, but then adding a dotted circle in the middle unless I add the above decomposition code.

[behdad]

Thanks Zachary for the detailed report. The USE spec clearly says it is up to the font designer to split the glyph if Unicode does not define a decomposition:

When a decomposition is not defined in UnicodeData.txt, it is up to the font developer to handle any required decomposition during GSUB processing.

However, the real question is, what does Windows do? If you can test and report that, I can adjust our implementation. Thanks!

[punchcutter]

@behdad Windows 10 splits the vowels in Edge and BabelPad. Yes, the USE spec says the font designer should do if it's not defined in UnicodeData.txt, but these two ARE defined in UnicodeData.txt:

1112E;CHAKMA VOWEL SIGN O;Mn;0;NSM;11131 11127;;;;N;;;;;
1112F;CHAKMA VOWEL SIGN AU;Mn;0;NSM;11132 11127;;;;N;;;;;

[behdad]

Ah, ok. Donno why we missed that before. At any rate, if they are in Unicode, they get applied in USE. Perhaps it's not working because we are not prohibiting recompose. See #382

[behdad]

@punchcutter can you try copying compose_indic() into USE shaper and see if that fixes the issue for you?

[punchcutter]

@behdad I copied decompose_indic to the USE shaper (like I wrote above) and that fixes the issue.

[behdad]

@behdad I copied decompose_indic to the USE shaper (like I wrote above) and that fixes the issue.

I understand. But I'm asking if you can copy compose_indic(), NOT decompose_indic(). Thanks!

[punchcutter]

compose_use is already in there, though. The only difference I see between compose_use and compose_indic is that indic has one extra exception for Bengali. Otherwise it's the same.

[behdad]

compose_use is already in there, though. The only difference I see between compose_use and compose_indic is that indic has one extra exception for Bengali. Otherwise it's the same.

Humm. You're right indeed. Ok, needs debugging. Will get there eventually... Thanks for checking.

[punchcutter]

Actually, it looks like the decomposition is happening and then the bottom mark is applied to the base, but the top mark gets a dotted circle instead of being applied to the base.

[jfkthame]

I think the issue here arises because the Unicode-defined decomposition:

1112E => 11131, 11127

where U+11131 is a below mark, and U+11127 is above, doesn't match the expected ordering of marks according to the USE spec[1]:

[< R | CS >] < B | GB > [VS] (CMAbv)* (CMBlw)* (< H B | SUB > [VS] (CMAbv)* (CMBlw)*)*
[MPre] [MAbv] [MBlw] [MPst]
(VPre)* (VAbv)* (VBlw)* (VPst)*
(VMPre)* (VMAbv)* (VMBlw)* (VMPst)*
(FAbv)* (FBlw)* (FPst)* [FM]

Therefore, the machine doesn't accept the trailing U+11127 (after decomposition) as part of the same cluster, but treats it as a separate broken cluster.

Note that if you try hb-shape with the input 11107,11131,11127, it will insert a dotted circle before 11127; while if you try 11107,11127,11131 it shapes successfully.

It's unfortunate that the canonical decomposition defined in Unicode disagrees with the cluster ordering expected by OpenType. :-(

[1] https://www.microsoft.com/typography/OpenTypeDev/USE/intro.htm#clustervalidation