Received mail may be truncated or its body may become empty when creating ticket from mail

[Jamesits]

Alpine's iconv don't have all the LUT preset; thus if osTicket received mails from these encodings (e.g. GB18030, which is still common in China Mainland), iconv conversion will fail, causing osTicket to treat source text as UTF-8 encoding, thus truncating anything after the first non-ASCII character.

Potential fix: either rebuild iconv with all the LUTs (I didn't try), or remove iconv support and use mbstring instead (I'm using this).

Note: in the develop branch of osTicket, the encoding function moved from Charset::transcode() to Format::encode(),so the patch I provided won't work with the next version.

[cleiter]

I used your second solution (mbstring) which worked fine until now. But I just got a mail with an utf-8 smiley and it truncated the message again.

[Jamesits]

@cleiter I haven't tried emojis. What's your database charset?

[cleiter]

Everything is UTF-8. Pretty sure the problem is in PHP. I probably have to go with the iconv solution to fix this once and for all... Also see docker-library/php#240

[cleiter]

So i tried that (built my own docker image) and actually it did not improve things. I investigated further and saw that the correct type in mysql is not utf8 but utf8mb4.

I did a quick:

set sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
alter table ost_thread_entry convert to character set utf8mb4 collate utf8mb4_unicode_ci;

(the first thing removes NO_ZERO_IN_DATE,NO_ZERO_DATE otherwise it won't let me update the table)

and now I get ???? instead of an emoji but the text is not truncated anymore.

Still not a perfect solution but hopefully it helps someone.

[Jamesits]

@cleiter utf8mb4 only affects 4-byte chars. Some emojis are 4-byte but most of them are 3-byte so they should be fine with utf8. However to use utf8mb4 you need to:

1. change default encoding in MySQL
2. set default encoding in PHP when establishing database session

Otherwise, it won't work.

[cleiter]

I did change the default encoding to utf8mb4 but that doesn't affect existing tables, hence the alter table query.

How would I set the mysql encoding in PHP for osticket?

[Jamesits]

@cleiter see https://stackoverflow.com/questions/16893035/using-utf8mb4-with-php-and-mysql

[cleiter]

OK thanks for your help :) I'll live with it as it is.

For reference, this was indeed a 4 byte smiley: 😊 (f09f 988a)

I would be super happy if the base image took care of all these encoding issues.