UnicodeEncodeError: 'utf-8' codec can't encode character '\udcea' in fetchmail

[vvolas]

Impacted versions: 11
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

Steps to reproduce: Get email with weird characters :) Odoo fails to process it.

I attached original untouched email, with vim I tried to go to exactly that character in email file and near that point there is line:
"To jest wieloczê�ciowa wiadomo�æ w formacie MIME." which I think causes problems.

Odoo error log(debug):
2019-08-06 10:56:11,128 406950 INFO gs odoo.addons.mail.models.mail_thread: Routing mail from "XXX" <XXX@XX.pl> to "AAAA" <ooo@XXX.net> with Message-Id <FAB15A96985E43F4967BD582B0578FD4@userKomputer>: direct reply to msg: model: crm.lead, thread_id: 5406, custom_values: None, uid: 1 2019-08-06 10:56:11,134 406950 INFO gs odoo.addons.fetchmail.models.fetchmail: Failed to process mail from imap server ooo@XXX.net. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/odoo/addons/fetchmail/models/fetchmail.py", line 179, in fetch_mail res_id = MailThread.with_context(**additionnal_context).message_process(server.object_id.model, data[0][1], save_original=server.original, strip_attachments=(not server.attach)) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1307, in message_process thread_id = self.message_route_process(msg_txt, msg, routes) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1235, in message_route_process new_msg = thread.message_post(subtype=subtype, partner_ids=partner_ids, **message_dict) File "/opt/odoo/addons_git/musu/bt_disable_partner_as_follower/models/mail_thread.py", line 164, in message_post attachment_ids = self._message_post_process_attachments(attachments, kwargs.pop('attachment_ids', []), values) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1751, in _message_post_process_attachments content = content.encode('utf-8') UnicodeEncodeError: 'utf-8' codec can't encode character '\udcea' in position 2569: surrogates not allowed 2019-08-06 10:56:11,141 406950 INFO gs odoo.addons.fetchmail.models.fetchmail: Fetched 1 email(s) on imap server ooo@XXX.net; 0 succeeded, 1 failed.

Expected behavior: Email gets parsed.

origininal_email

[flotho]

Hi @vvolas ,

We have the same issu on our fresh v12.
In our case, the point is with python3 and decoding datas from iso-8859-15 .
Here is the part utf-8 don't succeed to convert

Hi @Yenthe666 , could you recommend us somebody to ping .

Cordialement

[Yenthe666]

Most likely @tde-banana-odoo as he's the mail guy

[flotho]

Thanks @Yenthe666 ,

@tde-banana-odoo , is there any global conf we could set to use defensive on this kind of issue ?

[vvolas]

Email parsing errors are actually quite common, I even wrote module/hack which would register task in odoo of email parsing error, because now if it fails user has no idea that he got reply...

[flotho]

@vvolas ,
I'm interested in what you proposed.

Regards

[gglazer]

Hello Guys,
what is the solution? @tde-banana-odoo
Have you solved the problem? @vvolas @flotho
I got the same error with fresh Odoo v14.

[tde-banana-odoo]

Hello,

Thanks for pointing it out. We will have a look at it.

Thanks,

[gglazer]

Thanks!

To clarify the problem the log details:

2021-06-21 21:07:21,071 1 INFO erp odoo.addons.fetchmail.models.fetchmail: start checking for new emails on imap server Incoming Server - Support 2021-06-21 21:07:21,295 1 INFO erp odoo.addons.fetchmail.models.fetchmail: Failed to process mail from imap server Incoming Server - Support. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/odoo/addons/fetchmail/models/fetchmail.py", line 179, in fetch_mail res_id = MailThread.with_context(**additionnal_context).message_process(server.object_id.model, data[0][1], save_original=server.original, strip_attachments=(not server.attach)) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1129, in message_process msg_dict = self.message_parse(message, save_original=save_original) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1469, in message_parse msg_dict.update(self._message_parse_extract_payload(message, save_original=save_original)) File "/usr/lib/python3/dist-packages/odoo/addons/mail/models/mail_thread.py", line 1254, in _message_parse_extract_payload attachments.append(self._Attachment('original_email.eml', message.as_string(), {})) File "/usr/lib/python3.7/email/message.py", line 968, in as_string return super().as_string(maxheaderlen=maxheaderlen, policy=policy) File "/usr/lib/python3.7/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/python3.7/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.7/email/generator.py", line 195, in _write self._write_headers(msg) File "/usr/lib/python3.7/email/generator.py", line 222, in _write_headers self.write(self.policy.fold(h, v)) File "/usr/lib/python3.7/email/policy.py", line 183, in fold return self._fold(name, value, refold_binary=True) File "/usr/lib/python3/dist-packages/odoo/addons/base/models/ir_mail_server.py", line 49, in _fold return super()._fold(name, value, *args, **kwargs) File "/usr/lib/python3.7/email/policy.py", line 213, in _fold return self.header_factory(name, ''.join(lines)).fold(policy=self) File "/usr/lib/python3.7/email/headerregistry.py", line 258, in fold return header.fold(policy=policy) File "/usr/lib/python3.7/email/_header_value_parser.py", line 144, in fold return _refold_parse_tree(self, policy=policy) File "/usr/lib/python3.7/email/_header_value_parser.py", line 2652, in _refold_parse_tree part.ew_combine_allowed, charset) File "/usr/lib/python3.7/email/_header_value_parser.py", line 2736, in _fold_as_ew ew = _ew.encode(first_part) File "/usr/lib/python3.7/email/_encoded_words.py", line 224, in encode bstring = string.encode(charset) UnicodeEncodeError: 'utf-8' codec can't encode characters in position 8-9: surrogates not allowed 2021-06-21 21:07:21,316 1 INFO erp odoo.addons.fetchmail.models.fetchmail: Fetched 1 email(s) on imap server Incoming Server - Support; 0 succeeded, 1 failed.

This error does not always occur.

[flotho]

sadly we have no solution for this.
Regards

[gglazer]

sadly we have no solution for this.
Regards

Thanks for your answer!
I hope @tde-banana-odoo will found the solution. I tried to find, but without success.

[gglazer]

Any update for this @tde-banana-odoo ?
This problem still exists.

[tde-banana-odoo]

Hello @gglazer ,

If your issue is critical, better open an OPW. Indeed github issues have no deadline as they are handled by R&D (and not the Bugfix team).

Cheers,

[anhvu-sg]

Currently, my client has the same issue and I try to fix it.
It doesn't happen usually, we are using feature create lead automatically. Sometimes the end-user sends an email to the mailbox and Odoo creates the Lead but there is no information in the content.
Here is what do I have in the log:

2021-11-03 15:22:00,098 16133 INFO production_db odoo.addons.fetchmail.models.fetchmail: 2021-11-03 15:22:00,935 16133 ERROR production_db odoo.addons.crm.models.crm_lead: crm.lead > message_new > email_cc : 公司 - abc <abc@myemail.com> 2021-11-03 15:22:01,201 16133 INFO production_db odoo.addons.fetchmail.models.fetchmail: Failed to process mail from pop server XXX Traceback (most recent call last): File "/opt/odoo/odoo/addons/fetchmail/models/fetchmail.py", line 205, in fetch_mail res_id = MailThread.with_context(**additionnal_context).message_process(server.object_id.model, message, save_original=server.original, strip_attachments=(not server.attach)) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1425, in message_process thread_id = self.message_route_process(msg_txt, msg, routes) File "/opt/odoo/odoo/addons/mass_mailing/models/mail_thread.py", line 44, in message_route_process return super(MailThread, self).message_route_process(message, message_dict, routes) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1353, in message_route_process new_msg = thread.message_post(**post_params) File "/opt/odoo/odoo/addons/rating/models/mail_thread.py", line 14, in message_post message = super(MailThread, self).message_post(**kwargs) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1895, in _message_post_process_attachments content = content.encode('utf-8') UnicodeEncodeError: 'utf-8' codec can't encode character '\udcb3' in position 5563: surrogates not allowed 2021-11-03 15:22:01,293 5567 DEBUG production_db odoo.modules.registry: Multiprocess signaling check: [Registry - 587 -> 587] [Cache - 62380 -> 62380]

Maybe it's related to Latin, Mailer, attachment ...

@gglazer can you share with me your full email content?.

[vvolas]

This bug is in python mail or forgot how it was called library. I remember doing hackaround and updating files in python lib, but I no longer work there :)

…

On Thu, 4 Nov 2021, 10:39 Anh Vu, ***@***.***> wrote: Currently, my client has the same issue and I try to fix it. It doesn't happen usually, we are using feature create lead automatically. Sometimes the end-user sends an email to the mailbox and Odoo creates the Lead but there is no information in the content. Here is what do I have in the log: 2021-11-03 15:22:00,098 16133 INFO production_db odoo.addons.fetchmail.models.fetchmail: 2021-11-03 15:22:00,935 16133 ERROR production_db odoo.addons.crm.models.crm_lead: crm.lead > message_new > email_cc : 公司 - abc ***@***.***> 2021-11-03 15:22:01,201 16133 INFO production_db odoo.addons.fetchmail.models.fetchmail: Failed to process mail from pop server XXX Traceback (most recent call last): File "/opt/odoo/odoo/addons/fetchmail/models/fetchmail.py", line 205, in fetch_mail res_id = MailThread.with_context(**additionnal_context).message_process(server.object_id.model, message, save_original=server.original, strip_attachments=(not server.attach)) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1425, in message_process thread_id = self.message_route_process(msg_txt, msg, routes) File "/opt/odoo/odoo/addons/mass_mailing/models/mail_thread.py", line 44, in message_route_process return super(MailThread, self).message_route_process(message, message_dict, routes) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1353, in message_route_process new_msg = thread.message_post(**post_params) File "/opt/odoo/odoo/addons/rating/models/mail_thread.py", line 14, in message_post message = super(MailThread, self).message_post(**kwargs) File "/opt/odoo/odoo/addons/mail/models/mail_thread.py", line 1895, in _message_post_process_attachments content = content.encode('utf-8') UnicodeEncodeError: 'utf-8' codec can't encode character '\udcb3' in position 5563: surrogates not allowed 2021-11-03 15:22:01,293 5567 DEBUG production_db odoo.modules.registry: Multiprocess signaling check: [Registry - 587 -> 587] [Cache - 62380 -> 62380] Maybe it's related to Latin, Mailer ... @gglazer <https://github.com/gglazer> can you share with me your full email content?. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35485 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALBY2FIAUNEHENJBSWNSBTUKJBF5ANCNFSM4IJVKY7A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[bosd]

In 2023 the problem still exsists. Any update on this?