Not all emoji sequences recommended for general interchange (RGI) cluster

[rsheeter]

In testing emoji rgi clustering it seems polar bears and regional flags don't cluster. https://github.com/rsheeter/hb-emoji-clusters/blob/main/try_shape-stdout.txt has a list. For context, I tested all the sequences in https://unicode.org/Public/emoji/14.0/emoji-test.txt, which enumerates emoji sequences.

IIUC per Mark Davis all emoji should be grapheme clusters. I thought that would mean HB would cluster them but seemingly not. I see discussion on #2265 where a fix to make it so was discussed.

In the event emoji were to span multiple files it would help Chrome itemization if emoji rgi consistently formed clusters. If it's desirable that they don't I'd appreciate if someone could ELI5.

[behdad]

Regional-indicators are Complicated(TM) as seen in #2265.

Polar bear is the weirdest thing I've seen in Unicode: 🐻 U+1F43B, ZWJ U+200D,❄ U+2744, FE0F
️
This is our code implementing the grapheme logic:

harfbuzz/src/hb-ot-shape.cc

Lines 466 to 517 in bd5502f

	hb_set_unicode_props (hb_buffer_t *buffer)
	{
	/* Implement enough of Unicode Graphemes here that shaping
	* in reverse-direction wouldn't break graphemes. Namely,
	* we mark all marks and ZWJ and ZWJ,Extended_Pictographic
	* sequences as continuations. The foreach_grapheme()
	* macro uses this bit.
	*
	* https://www.unicode.org/reports/tr29/#Regex_Definitions
	*/
	unsigned int count = buffer->len;
	hb_glyph_info_t *info = buffer->info;
	for (unsigned int i = 0; i < count; i++)
	{
	_hb_glyph_info_set_unicode_props (&info[i], buffer);
	/* Marks are already set as continuation by the above line.
	* Handle Emoji_Modifier and ZWJ-continuation. */
	if (unlikely (_hb_glyph_info_get_general_category (&info[i]) == HB_UNICODE_GENERAL_CATEGORY_MODIFIER_SYMBOL &&
	hb_in_range<hb_codepoint_t> (info[i].codepoint, 0x1F3FBu, 0x1F3FFu)))
	{
	_hb_glyph_info_set_continuation (&info[i]);
	}
	#ifndef HB_NO_EMOJI_SEQUENCES
	else if (unlikely (_hb_glyph_info_is_zwj (&info[i])))
	{
	_hb_glyph_info_set_continuation (&info[i]);
	if (i + 1 < count &&
	_hb_unicode_is_emoji_Extended_Pictographic (info[i + 1].codepoint))
	{
	i++;
	_hb_glyph_info_set_unicode_props (&info[i], buffer);
	_hb_glyph_info_set_continuation (&info[i]);
	}
	}
	#endif
	/* Or part of the Other_Grapheme_Extend that is not marks.
	* As of Unicode 11 that is just:
	*
	* 200C ; Other_Grapheme_Extend # Cf ZERO WIDTH NON-JOINER
	* FF9E..FF9F ; Other_Grapheme_Extend # Lm [2] HALFWIDTH KATAKANA VOICED SOUND MARK..HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK
	* E0020..E007F ; Other_Grapheme_Extend # Cf [96] TAG SPACE..CANCEL TAG
	*
	* ZWNJ is special, we don't want to merge it as there's no need, and keeping
	* it separate results in more granular clusters. Ignore Katakana for now.
	* Tags are used for Emoji sub-region flag sequences:
	* https://github.com/harfbuzz/harfbuzz/issues/1556
	*/
	else if (unlikely (hb_in_range<hb_codepoint_t> (info[i].codepoint, 0xE0020u, 0xE007Fu)))
	_hb_glyph_info_set_continuation (&info[i]);
	}
	}

For emoji, we append any ZWJ,Extended_Pictograph sequence to the previous cluster. U+2744 SNOWFLAKE is in that list. So I expect that we handle this sequence correctly. Let me check.

[behdad]

Maybe we should add an equivalent of your code to the test suite.

[behdad]

Oops... Bad bug in emoji table generator...

[behdad]

@rsheeter Can you please rerun your script against master and attach conclusion?

[rsheeter]

It would appear you have rescued the polar bear! rsheeter/hb-emoji-clusters@85bfd1f

[behdad]

Re the regional-indicator pairs: #2265 (comment)

[rsheeter]

#3018 probably fixes this;.

[behdad]

#3018 probably fixes this;.

Interesting to see if it actually does.

[rsheeter]

0 / 4702 failures, looks like it does :)

[drott]

Big thanks for doing this list, @rsheeter! And glad we found an actual issue here.

[khaledhosny]

This should be closed now, right?

[behdad]

Is fixed. Importing as a test would be nice.

[behdad]

Here's a slow way to get started. It's slow because we don't have --unicode-files...

$ echo x > null; curl https://www.unicode.org/Public/emoji/13.1/emoji-zwj-sequences.txt | cut -d';' -f1 | cut -d'#' -f1 | grep . | while read line; do ./hb-shape --font-file=null --unicodes="$line" --no-positions | grep -q '=[^0]' && ./hb-shape --font-file=null --unicodes="$line" --no-positions --show-text --show-unicode; done

Maybe just make gen-emoji-table.py emit a test file somewhere in test/shaping/data/in-house/tests. The need for non-empty null font file is also because of #2567