Unicode math italics/bold truncates text notes

[fgnievinski]

A reviewer copied content from an article (typeset externally), resulting in formatted text such as "𝑚𝑒𝑡𝑒𝑟𝑠" (contrast: meters and meters).

When such text is pasted in a TineMCE box, it's displayed correctly, but saving the text is not possible:

I believe these Latin italics characters belong to the Unicode Mathematical Alphanumeric Symbols block.
Maybe the database doesn't support storing such so extended Unicode?

Slightly related issue: #2564

[asmecher]

@fgnievinski, OJS should support the full Unicode set. Can you please include the following information:

• What version of OJS are you using?
• Does anything appear in the PHP error log from the time the request was made to save the form?

[fgnievinski]

I was using OJS 3.3.0.8 in an installation where I don't have access to the logs (it's a hassle to get IT to do anything, sorry).

but I've just tried in the public demo/testdrive installation (also v.3.3.0.8) -- maybe the logs could be retrieved more easily?

pasting "A𝐵C" (with math italics only in the middle char) results in truncated string starting at the offending char, so only A is stored.

• in a non-TinyMCE field: Workflow > Submission > Submission Files > 🠞🠟 > More Information > Notes > Add Note
  

• in a TinyMCE field: Workflow > Review > Round 1 > Reviewers > 🠞🠟 > Editorial Notes > OK
  

I've tried to replicate at my local installation and I'm still unable to save. The only difference seemed to be the primary language, so I've changed the primary language in the testdrive but was still able to save the notes, although truncation persists.

So the reproducible problem is: text gets truncated after math italics characters.

[jonasraoni]

As a note... I've had some encoding issues when using the utf8 charset in MySQL, but they were gone after I updated the database to use the newer utf8mb4.
If you're going to try, ensure the database, tables and also the columns are all using the same charset and collation.

[fgnievinski]

As a note... I've had some encoding issues when using the utf8 charset in MySQL, but they were gone after I updated the database to use the newer utf8mb4. If you're going to try, ensure the database, tables and also the columns are all using the same charset and collation.

thanks for the tip. the installation has been recently restored after been invaded, so probably there are some lose ends.

I'll change the issue description to focus on the reproducible problem.

[jonasraoni]

Anyway, every application will present some kind of weird behavior when faced with the strings in this list https://github.com/minimaxir/big-list-of-naughty-strings, so perhaps you've found one of them :)

[fgnievinski]

just to tell a sad story: the reviewer submitted their review but didn't notice it was truncated then the author ended up missing half of the reviewer's comments, who was not very pleased. :-|

[NateWr]

I was able to reproduce this on the OJS3 test drive install but not locally.

Locally, I am running psql (PostgreSQL) 12.9 (Ubuntu 12.9-0ubuntu0.20.04.1) and this is the character encoding:

ojs_330=# SHOW SERVER_ENCODING;
 server_encoding 
-----------------
 UTF8

It seems this is related to the deployment in some way. If this requires a change to the default or recommended database configuration, can someone propose a change?

[jonasraoni]

@NateWr I've just checked in my environment and my previous comment is enough to address the issue. Just forgot to add that the charset must be configured in the config.inc.php:

[i18n]
connection_charset = utf8mb4

[database]
collation = utf8mb4_general_ci

As I've already created an issue to address this configuration, I'll close this one.

[NateWr]

It's interesting, though, because these are my config settings and I didn't have the problem. Is it because I'm on postgres?

[i18n]

; Default locale
locale = en_US

; Client output/input character set
client_charset = utf-8

; Database connection character set
; Must be set to "Off" if not supported by the database server
; If enabled, must be the same character set as "client_charset"
; (although the actual name may differ slightly depending on the server)
connection_charset = utf8

; Database storage character set
; Must be set to "Off" if not supported by the database server
database_charset = utf8

Also, I don't have [database]collation at all in my config.inc.php. I see it in the template though. My config is probably from a long time ago...

[jonasraoni]

Yeah, it happens only in MySQL. The old utf8 charset supports at maximum characters composed by 3 bytes, but we happen to have characters with 4 bytes 😁

PostgreSQL supports the 4 bytes pattern, but it will also fail if you try to insert an invalid UTF-8 sequence.