Choices do not work with ❌ emoji

[cesarvarela]

Versions

"botbuilder": "^4.3.4",
v10.14.0
OSX

Describe the bug

Chioce prompt doesn't work with the ❌ emoji, and for example 💚 does work, in the same list of options.

To Reproduce

        const prompt = MessageFactory.carousel(card)
        const choices = CardFactory.actions(['❌', 'Skip', '💚'])

        return await step.prompt('promptlikes', { prompt, choices })

Expected behavior

Should work with every emoji

Additional context

Debugging a bit I've found that the defaultTokenizer doesn;t recognize the emoji and thinks its a breaking char?

[bug]

[johnataylor]

The tokenizer behavior of splitting on some emoji is described here: https://github.com/Microsoft/botbuilder-dotnet/issues/706

[johnataylor]

Yes this is correct the ❌ U+274C is being recognized as a breaking character (like punctuation) and the 💚 U+1F49A isn't.

Emoji's are a little more complex than the current code assumes, simply working on code-points is not enough, the full emoji will often be multiple code-points. The length of the emoji is variable. In some cases like the flag of Wales up to 7 code-points (and in this case 14 characters.)

Refer to https://unicode.org/emoji/charts/full-emoji-list.html

So this amounts to quite a heavy redesign of this code. Moving to the next milestone.

However the Tokenizer is replaceable. The behavior we are talking about here is just the behavior of the default tokenizer. If you have an urgent need for slightly different tokenizing code you might consider plugging in a custom tokenizer.

[cleemullins]

pushing to 4.7 due to resource constraints in 4.6.

[cleemullins]

@johnataylor - any update on this?

[johnataylor]

We are not planning enhanced emoji support in R9 moving to R10.

[johnataylor]

closing and tracking this with the issue in borframework-sdk