Rich Text Editor - Emoji Picker Plugin

[abrookbanks]

😜😀😊😃 becomes ????????????????

I've tried utf8mb4_unicode to no avail..

[bhsmither]

What is an "HTML Emoji" (as opposed to any other kind of emoji)?

The version of CKeditor shipped with CC653 is 4.12.1, while the latest version is 4.23-lts (4.22.1 as open source).

Maybe an upgrade, using the latest emoji plugin might get these other emojis.

https://ckeditor.com/docs/ckeditor4/latest/features/emoji.html

It is interesting to note on that page:

Please note that the emoji feature uses system fonts to render emojis. CKEditor 4 does not provide them on its own which means emojis availability depends on your operating system.

[abrookbanks]

e.g. 😀 = 😀

[abrookbanks]

What is an "HTML Emoji" (as opposed to any other kind of emoji)?

The version of CKeditor shipped with CC653 is 4.12.1, while the latest version is 4.23-lts (4.22.1 as open source).

Maybe an upgrade, using the latest emoji plugin might get these other emojis.

https://ckeditor.com/docs/ckeditor4/latest/features/emoji.html

It is interesting to note on that page:

Please note that the emoji feature uses system fonts to render emojis. CKEditor 4 does not provide them on its own which means emojis availability depends on your operating system.

Yes I tried that but the version was an issue. I think there was a block upgrading from 4.12.1 but I'll look into it. I don't see why emojis can't just be added via character map copy / paste.

[bhsmither]

I just tried that 😀 and it worked for me (CC652, Firefox 115.4).

[abrookbanks]

What encoding is your database set to?

[bhsmither]

Well, looking at the database directly, CubeCart_inventory, 'description' column, I have:

<p>This is the main &#39;copy&#39; for the &#128512; product.</p>

Does the character encoding or collation really matter? It is the job of the browser to render the HTML entity code.

The CubeCart database default collation: utf8_unicode_ci
The database server:
  collation_server: utf8_general_ci
  collation_database: utf8_general_ci
  collation_connection: utf8mb4_general_ci
  character_set_system: utf8
  character_set_server: utf8
  character_set_results: utf8mb4
  character_set_database: utf8
  character_set_connection: utf8mb4
  character_set_client: utf8mb4

[abrookbanks]

Well no I didn't think so either. Hmmmm. It doesn't work on my development store. It's got to be an encoding issue somewhere..

[bhsmither]

Just to be clear, where are the question marks appearing initially?

After entering the HTML entity code in Source mode, then switching back to WYSIWYG view?

Or after saving and viewing the content in the storefront?

[abrookbanks]

On reload.

…

On Fri, 10 Nov 2023, 19:52 Brian Smither, ***@***.***> wrote: Just to be clear, where are the question marks appearing initially? After entering the HTML entity code in Source mode, then switching back to WYSIWYG view? Or after saving and viewing the content in the storefront? — Reply to this email directly, view it on GitHub <#3420 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCPWYZUUYBNFYPIAJ5DQCTYD2AWLAVCNFSM6AAAAAA7F4XOLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGM2TCMZQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bhsmither]

Hmm... using Save & Reload button worked before, but this last time I got question marks.

[bhsmither]

And the database now has four question marks. But I cannot tell if they are ctual question marks or the default character to show for high-ascii bytes.

Well, PHP received it in POST as hex 9F 98 80 (my editor complained about invalid utf8 sequences).

CubeCart's debug shows in POST['description'] the actual smiley image -- browser is rendering hex F0 9F 98 80.

CubeCart sent the sequence hex F0 9F 98 80 in the database update query.

So, assuming in some cases CKeditor converts the HTML entity code to actual UTF8MB4 code points, I will test with setting the collation for CubeCart_inventory, description column to utf8mb4_general_ci.

I will also research to see if there is a bug report about CKeditor converting HTML entities to utf8 sequences when it shouldn't.

[bhsmither]

Cannot change 'description' to utf8mb4_general_ci as it is a member of a FULLTEXT index. I wonder if there is a solution to that.

[bhsmither]

I'm convinced its an old, unsolved bug in CKeditor:

https://stackoverflow.com/questions/36574026/disable-conversion-of-html-entities-in-ckeditor

[abrookbanks]

[abrookbanks]

Upgrade with new plugins fixed it.

[abrookbanks]

This still doesn't work..

[abrookbanks]

The issue was the database connection character set. But what happens if the database isn't ready for utf8mb4.

[abrookbanks]

Relates to #3425

[bhsmither]

What I would like to see is that CKEditor does not convert html entities into the Unicode byte sequence. Haven't found that setting, yet.

[abrookbanks]

Relates to #3425