Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving "special" emoji in text fields causes 'Control character error, possibly incorrectly encoded.' #2725

Closed
sevfurneaux opened this issue Apr 11, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@sevfurneaux
Copy link

commented Apr 11, 2018

When saving "special" emoji within text fields (such as 🦄), causes this error:

Invalid Argument – yii\base\InvalidArgumentException
Control character error, possibly incorrectly encoded.

This error is raised in in /app/vendor/yiisoft/yii2/helpers/BaseJson.php and /app/vendor/craftcms/cms/src/services/EntryRevisions.php.

At line 390 in EntryRevisions.php:

$result['data'] = Json::decode($result['data']);

After further investigation looking in the database, this is an issue specifically with the entryversions table and what gets saved in the data column when a "special" emoji is used.

You can see a comparison below, with the data column content showing an entry with normal text and an example when I saved the entry with a 🦄:

Normal Data

{"typeId":"14","authorId":"1","title":"This is a test entry","slug":"this-is-a-test-entry","postDate":1523459184,"expiryDate":null,"enabled":true,"newParentId":null,"fields":{"181":"Normal content"}}

When saved with 🦄

{"typeId":"14","authorId":"1","title":"This is a test emoji","slug":"this-is-a-test-emoji","postDate":1523459260,"expiryDate":null,"enabled":true,"newParentId":null,"fields":{"181":"

You can see above, that the data was not saved correctly when the special emoji is used. It cuts off at {"181":" and therefore causes a JSON decoding error.

@brandonkelly

This comment has been minimized.

Copy link
Member

commented Apr 11, 2018

I’m not able to even save an entry with a 🦄 when using MySQL (get a SQL error), and not able to reproduce the the exception when using PostgreSQL. Have you made any changes to your environment to get past this error in MySQL?

SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\xA6\x84' for column 'field_<field_handle>' at row 1

@sevfurneaux

This comment has been minimized.

Copy link
Author

commented Apr 11, 2018

I've managed to recreate this in two different environments: Nanobox (locally and on a staging site) and also in MAMP Pro 4.2.

In both environments when saving the 🦄, it produced the malformed data in the data column of the entryversions table.

@brandonkelly

This comment has been minimized.

Copy link
Member

commented Apr 13, 2018

We fixed this at the root, by adding a validation error for textual custom fields that contain emojis (or any other 4+ byte characters), if the DB doesn't support 4+ byte characters. (Any characters that are converted to HTML entities or emoji shortcodes by the field type before being saved won’t be affected.)

@sevfurneaux

This comment has been minimized.

Copy link
Author

commented Apr 13, 2018

Great, thanks!

@bossanova808

This comment has been minimized.

Copy link

commented May 30, 2019

Hmm, I have jsut hit this issue with a customer adding some order notes (to some line item custom data) with an emoji in it.

I am wondering what the best fix is in this case - db end (unicode mb4?) - js (strip emjois - but inevitably people will be putting them in and expect them to work I expect) - ... or somewhere else?

@brandonkelly Any tips on this?

@brandonkelly

This comment has been minimized.

Copy link
Member

commented May 30, 2019

@bossanova808 Can you repost on the Commerce repo?

@bossanova808

This comment has been minimized.

Copy link

commented May 30, 2019

Issue here - craftcms/commerce#886 (comment)

I commented on the fix too, because it's not complete for my use case yet: craftcms/commerce@11c5dc2 - but maybe it's too hard in that case? Not sure how universal you want to be with accepting this data...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.