Some Emojis break Notes

[JamHaven]

When using certain emojis (e.g. 😂 via Windows + . ) in a note, the note can no longer be changed and always reverts back to the moment the emoji was written.

Steps to reproduce:

1. Write Text "This is a beautiful day"
2. Add Emoji (tested via Windows + . ) "😂"
3. Text now Reads "This is a beautiful day 😂". Try to write something new (e.g. "No! This day is awful.")
4. Text gets reverted to "This is a beautiful day 😂"
5. Try to delete the message
6. Text gets reverted to "This is a beautiful day 😂"

Note: ☠️ works fine, and 🦖 behaves differently, by deleting more parts of the text and does not break the notes.

[MWedl]

We also encountered this bug during testing and it should already be fixed on version 2024.40. Could you please check if you are using the latest version or an older version. Please let us know if the bug persists for some emojis on the latest version.

The problem is that JavaScript (used in frontend) and Python (used in backend) handle unicode characters in strings differently. JS handles strings internally as UTF-16 while Python uses UTF-32. The impact of the discrepancy is that strings containing unicode characters have different string lengths in frontend and backend. This breaks string indexing and slicing for collaborative editing, since text change positions are transmitted as string indices. We tried to solve the problem by calculating string positions in the backend like JS does.

Unicode is quite fun. For more info on this topic see https://hsivonen.fi/string-length/

[JamHaven]

Thank you, can confirm the bug is fixed in the latest version.

I also love to have some fun with unicode hehe