Emoji ZWJ Sequences are not grouped as single cluster #217

drott · 2016-01-20T11:24:59Z

Emoji ZWJ Sequences are not recognized/grouped into a single cluster when shaping them with a font that does not have glyphs for them. This makes emoji conflict with the shaper-driven segmentation approach we take in Chrome. If the page for example specifies Arial as a first font, the ZWJs get shaped with spaces, and the individual person emoji remain, which then get shaped with the fallback emoji font, but the run is already broken up by the zero width spaces taken from Arial, example:


$ ./hb-shape /Library/Fonts/Arial.ttf "👩‍👩‍👧‍👧 "
[.notdef=0+1536|space=1+0|.notdef=2+1536|space=3+0|.notdef=4+1536|space=5+0|.notdef=6+1536|space=7+569]

So here the Emoji glyphs are not found, but the ZWJs are shaped and rendered as zero width space glyphs from Arial.

At the moment, this could only be overriden in Blink by always explicitly prioritizing the system Emoji font over the fonts specified on the page for Emoji sequences.

Instead, however, I would propose to fix this by making them the same cluster and putting one .notdef for the whole emoji ZWJ sequence. Actually, probably similarly we need to think about the +Keycap and double-regional indicator sequences (as in Emoji flags).

Related to, but not duplicate of issue #179, I believe. CC @roozbehp

The text was updated successfully, but these errors were encountered:

behdad · 2016-01-20T11:54:46Z

Keycap is already handled in cluster-level=0 (which I think Chrome uses). I don't mind adding flags and emoji...

behdad · 2016-01-20T11:56:09Z

On the other hand, most clients should be using cluster-level=1 and do their own Unicode grapheme boundaries for fallback anyway.

khaledhosny · 2016-01-20T16:48:48Z

Right which is sorta like what LibreOffice is doing, if a character is unsupported by the first font (0 GID) the whole Unicode grapheme cluster is reshaped, not just that character.

drott · 2016-02-05T13:31:12Z

Any new thoughts on this? Or is the right solution to use a different unit of fallback in Blink? (Edit:) It would be good if we could have HarfBuzz grouping them as a temporary solution before changing the fallback unit in Blink.

behdad · 2016-02-22T09:22:09Z

Because it's convenient, I'm going to combine ZWJ/ZWNJ with their previous cluster in cluster-level=0, as that's what the Unicode grapheme boundary algorithm does.
http://unicode.org/reports/tr29/

roozbehp · 2016-02-22T18:47:34Z

Behdad, UAX #29 is just about to change for ZWJ for Unicode 9 based on http://www.unicode.org/L2/L2016/16011r3-break-prop-emoji.pdf (which the UTC approved in their last meeting).

Also, I'm not sure I understand the exact change, but wouldn't this affect a sequence such as "Space, ZWJ, Arabic letter Ain"?

Reopening just in case.

behdad · 2016-02-23T01:34:45Z

Behdad, UAX #29 is just about to change for ZWJ for Unicode 9 based on http://www.unicode.org/L2/L2016/16011r3-break-prop-emoji.pdf (which the UTC approved in their last meeting).

Right. We can add more stuff here but prefer to keep it minimal.

Also, I'm not sure I understand the exact change, but wouldn't this affect a sequence such as "Space, ZWJ, Arabic letter Ain"?

Affects, yes. But it's no worse than before. If the layout system doesn't pass full context to HarfBuzz, then each ZWJ will end up in one run if the two sides are separated. Which one it will become part of is arbitrary. That said, I might do something smarter based on surroundings. But won't fix A,ZWJ,Ain.

Reopening just in case.

behdad · 2016-02-23T02:05:52Z

Closing for now. Will review additional cases separately.

Overview of changes leading to 1.2.1 Friday, February 23, 2016 ==================================== - CoreText: Fix bug with wrong scale if font scale was changed later. libass/libass#212 - CoreText: Drastically speed up font initialization. - CoreText: Fix tiny leak. - Group ZWJ/ZWNJ with previous syllable under cluster-level=0. harfbuzz/harfbuzz#217 - Add test/shaping/README.md about how to add tests to the suite. Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>

Overview of changes leading to 1.2.1 Friday, February 23, 2016 ==================================== - CoreText: Fix bug with wrong scale if font scale was changed later. libass/libass#212 - CoreText: Drastically speed up font initialization. - CoreText: Fix tiny leak. - Group ZWJ/ZWNJ with previous syllable under cluster-level=0. harfbuzz/harfbuzz#217 - Add test/shaping/README.md about how to add tests to the suite. (From OE-Core rev: 6dc2772) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Overview of changes leading to 1.2.1 Friday, February 23, 2016 ==================================== - CoreText: Fix bug with wrong scale if font scale was changed later. libass/libass#212 - CoreText: Drastically speed up font initialization. - CoreText: Fix tiny leak. - Group ZWJ/ZWNJ with previous syllable under cluster-level=0. harfbuzz/harfbuzz#217 - Add test/shaping/README.md about how to add tests to the suite. (From OE-core rev: 6dc27726c9da5104ea27ebf604cab6ef2037034d) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>

* freetype: Make optional, rename feature. Now, the feature for this is called `freetype` rather than `build-native-freetype`. This was never building FreeType but just saying to build with FreeType and to use `freetype-sys`. The default features were disabled for `freetype` and then enabling the `freetype-sys` feature, but that's exactly what the `freetype` crate does by default, so simplify by not trying to do it again. Partial fix for harfbuzz#184 (this doesn't handle the `coretext` side). * Add `directwrite` feature. This controls whether or not `winapi` is used to support the DirectWrite code on Windows. * Add `coretext` feature. This controls whether or not the CoreText APIs are supported on Apple platforms. Fixes harfbuzz#184. * Add features list to crate docs and READMEs. * Add notes to functions about feature required.

Overview of changes leading to 1.2.1 Friday, February 23, 2016 ==================================== - CoreText: Fix bug with wrong scale if font scale was changed later. libass/libass#212 - CoreText: Drastically speed up font initialization. - CoreText: Fix tiny leak. - Group ZWJ/ZWNJ with previous syllable under cluster-level=0. harfbuzz/harfbuzz#217 - Add test/shaping/README.md about how to add tests to the suite. (From OE-Core rev: 6dc27726c9da5104ea27ebf604cab6ef2037034d) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

behdad closed this as completed in 815bdd7 Feb 22, 2016

roozbehp reopened this Feb 22, 2016

behdad closed this as completed Feb 23, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emoji ZWJ Sequences are not grouped as single cluster #217

Emoji ZWJ Sequences are not grouped as single cluster #217

drott commented Jan 20, 2016

behdad commented Jan 20, 2016

behdad commented Jan 20, 2016

khaledhosny commented Jan 20, 2016

drott commented Feb 5, 2016

behdad commented Feb 22, 2016

roozbehp commented Feb 22, 2016

behdad commented Feb 23, 2016

behdad commented Feb 23, 2016

Emoji ZWJ Sequences are not grouped as single cluster #217

Emoji ZWJ Sequences are not grouped as single cluster #217

Comments

drott commented Jan 20, 2016

behdad commented Jan 20, 2016

behdad commented Jan 20, 2016

khaledhosny commented Jan 20, 2016

drott commented Feb 5, 2016

behdad commented Feb 22, 2016

roozbehp commented Feb 22, 2016

behdad commented Feb 23, 2016

behdad commented Feb 23, 2016