Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when events contain utf8 characters #36

Open
DanCech opened this issue Jul 9, 2021 · 1 comment
Open

Errors when events contain utf8 characters #36

DanCech opened this issue Jul 9, 2021 · 1 comment

Comments

@DanCech
Copy link
Contributor

DanCech commented Jul 9, 2021

If events contain emojis we end up with error messages like:

t=2021-07-09T17:28:25+0000 lvl=eror msg="failed to download file" workerId=7 createdAt=2021-07-09T15:00:00+0000 error="Error 1366: Incorrect string value: '\\xF0\\x9F\\x98\\x84 \"...' for column 'data' at row 1"

\xF0\x9F\x98\x84 is the utf8 representation of 😄 , and it appears that we need to update the database schema and connection character sets to properly handle utf8 characters.

@marefr
Copy link
Member

marefr commented Jul 9, 2021

Yes, I know 😄 I have a branch somewhere that changes the JSON decoding and don't fail on emojis. I will try to push that up and open a PR. Currently we don't parse the contents such as emojis hence we probably don't need to change the database schema now. Maybe later, but would require re-import everything - the solution is built for that in mind though, but takes quite some time 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants