-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Email parser creates a message object that can't be flattened as bytes. #83565
Comments
This is similar to https://bugs.python.org/issue32330 but is the opposite behavior. In that issue, the message couldn't be flattened as a string but could be flattened as bytes. Here, the message can be flattened as a string but can't be flattened as bytes. The original message was created by an arguably defective email client that quoted a message containing a utf8 encoded RIGHT SINGLE QUOTATION MARK and utf-8 encoded separately the three bytes resulting in The following interactive python session shows the issue.
|
Since you parsed it as a string it is not really legitimate to serialize it as bytes. (That will work if the input message only contains ascii, but not if it contains unicode). You'll get the same error if you replace the garbage with the "’". Using errors=replace is not crazy, but it hides the actual problem. Let's see what other people think :) In theory you could "fix" this by encoding the unicode using the charset specified by the container. I have no idea how complicated it will be do that, and it would be a new feature: parsing strings is specified to only work with ASCII input, currently. I put "fix" in quotes, because even if you make text parts like this example work, you still can't handle non-text 8bit mime parts. Is it worth doing anyway? Really, message_as_string and friends should just be avoided entirely, maybe even deprecated. |
This came about because of an actual situation in a Mailman 3 installation. I can't say for sure what the actual original message looked like, but it was received by Mailman's LMTP server and parsed with email.message_from_bytes(), so it clearly wasn't exactly like the message excerpt I posted in the report above. However, All I had to go by was the message object from the shunted pickle file created as a result of the exception. The message was processed by Mailman, but when Mailman's handler pipeline attempted to save it for the digest, it calls an instance of mailbox.MMDF to add the message to the mailbox accumulating messages for the digest, and that in turn calls the flatten method of an email.generator.BytesGenerator instance. and that's where the exception was thrown. Perhaps the suggested patch in #18056 doesn't address every possible case, and it can result in a slightly garbled message due to replacing 'invalid' characters, but in my case at least, it is much preferable to the alternative. |
Other Mailman3 installations are also encountering this issue. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/VQZORIDL5PNQ4W33KIMVTFTANSGZD46S/ |
If we can get an actual reproducer using message_as_bytes I'd feel more comfortable with the fix. I worry that there is some other bug this is exposing that should be fixed instead. |
message_from_bytes |
I've researched this further, and I know how this happens. The original message contains a text/html part (in my case, the only part) which contains a base64 or quoted-printable body which when decoded contains non-ascii. It is parsed correctly by email.message_from_bytes. It is then processed by Mailman's content filtering which retrieves html payload via
where part is the text/html part and ctype is 'utf-8' in this case. It then uses elinks, lynx or some other configured command to convert the html payload to plain text and that plain text still contains non-ascii. It then replaces the payload and sets the content type via
And this results in a message which can't be flattened as_bytes. The issue is set_payload() should encode the payload appropriately and in fact, it does if an appropriate charset is given, so this is our error in not providing a charset= argument to set_payload. Closing this and the corresponding PR. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: