Unilateral change to USE cluster model

[NorbertLindenberg]

Pull request #3479 changed the regular expression for the cluster model that's at the core of the Universal Shaping Engine. There's no indication in the pull request that this was discussed with anybody outside the HarfBuzz team, and there doesn't seem to be an issue filed against the USE specification.

I find this quite problematic. We have a big problem already in not having the cluster models for most complex script specified in any meaningful way. This has led to divergent implementations, as I've documented for Khmer and Devanagari. This in turn has led to problems with incorrect encodings of strings that enable spoofing and prevent successful search, as documented in Joshua Horton et al.: Spoof-Vulnerable Rendering in Khmer Unicode Implementations (ACIS2017) and Appendix 1 of Martin Hosken: Khmer Encoding Structure.

The Universal Shaping Engine has been a big improvement in this situation, as it provides a well-defined cluster model for Brahmic scripts based on Unicode data. This makes it much clearer how text should be encoded, and much more predictable when dotted circles will be inserted. I've used the USE cluster model to implement keyboards and AAT-based fonts, have worked with Unicode script proposal authors to define and verify the cluster model for scripts that they're proposing for encoding, and have recently submitted a proposal to correct the Javanese syllable structure in the Unicode Standard based on the USE cluster model. Javanese is one of the 18 scripts affected by pull request #3479, so HarfBuzz already does not conform to the proposed syllable structure.

I would appreciate if you could back out this change. We can then try and start a discussion on whether this change is appropriate to make in the specification and all implementations. Yes, I'm aware that Microsoft appears to have abandoned the specification – that is a problem we have to get addressed in some form.

[behdad]

Yes, I'm aware that Microsoft appears to have abandoned the specification – that is a problem we have to get addressed in some form.

How do you suggest to get that addressed then?

[NorbertLindenberg]

My preference would be that text rendering system implementors, font developers, and font tool developers together fix and improve the shaping engine specifications under the umbrella of the Unicode Consortium.

[dscorbett]

It would be a good thing for many people to agree on such changes. I always give rationale and examples in these pull requests so people can make properly informed disagreements, though I don’t think anyone has yet. I am worried that a Unicode committee would create lots of script-specific patterns as in L2/22-041, which is a bad idea for a universal shaping engine.

You are not opposed to all overrides of the USE spec in HarfBuzz, I think, so your objection is that the change affects too many scripts. You don’t necessarily think the changes were wrong per se, just that they should have been discussed more broadly. Have I understood right?

Reverting the change would break Saurashtra. Waiting for Microsoft takes too long. Waiting for Unicode to make a new shaping engine committee would take too long. Some sort of change was necessary. Furthermore, reverting the change would reintroduce a semi-accidental unilateral change to Grantha. Presumably you do not want that. What should we have done instead?

The most conservative change would be to revert everything, remove the HVM category, add a new category for U+A8B4 SAURASHTRA CONSONANT SIGN HAARU that is allowed in a virama_terminated_cluster, and add a new category for U+1134D GRANTHA SIGN VIRAMA that is a match for dependent_vowels. I dislike adding special cases for specific characters.

[behdad]

Agree with David that waiting for Microsoft or Unicode is not feasible.

[nedley]

My recollection is that this issue came up during the OpenType meeting hosted by Facebook back in 2019. The specific issue had to do with Sinhala U+0DDD: Andrew’s suggestion was to allow H between the V and VM segments of a standard cluster, which is what CoreText does. Sadly, it appears this never made it into the USE documentation.

[NorbertLindenberg]

I am worried that a Unicode committee would create lots of script-specific patterns as in L2/22-041, which is a bad idea for a universal shaping engine.

The pattern in L2/22-041 is derived from the documented cluster model of the USE, so it's not in conflict with it.

You are not opposed to all overrides of the USE spec in HarfBuzz, I think, so your objection is that the change affects too many scripts. You don’t necessarily think the changes were wrong per se, just that they should have been discussed more broadly. Have I understood right?

Correct. Change is often necessary as we learn more about scripts and their actual needs, but it should be standardized across implementations to ensure interoperability. I was quite surprised though when I found out a few weeks ago how many overrides HarfBuzz uses without bringing them to the attention of the maintainer of the USE overrides.

Agree with David that waiting for Microsoft or Unicode is not feasible.

My recollection is that this issue came up during the OpenType meeting hosted by Facebook back in 2019. The specific issue had to do with Sinhala U+0DDD: Andrew’s suggestion was to allow H between the V and VM segments of a standard cluster, which is what CoreText does. Sadly, it appears this never made it into the USE documentation.

I give up.

[nedley]

I’m happy to take the blame for not following up in making sure the documentation got updated, even if it isn’t mine to update. At least with approach taken by CoreText no new classes are needed to fix the most obvious problems.