Email parser creates a message object that can't be flattened

[msapiro]

BPO	27321
Nosy	@warsaw, @msapiro, @diekhans, @bitdancer, @soltysh, @matrixise, @csabella, @kyrias, @miss-islington, @edeca
PRs	bpo-27321: email: don't try to replace headers that aren't set #1977 bpo-27321 Fix email.generator.py to not replace a non-existent header. #18074 [3.9] bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074) #22796 [3.8] bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074) #22797
Files	bad_email: The problem message generator.patch: Suggested fix

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-19.23:02:01.155>
created_at = <Date 2016-06-14.18:52:52.474>
labels = ['type-bug', '3.8', 'expert-email', '3.10', '3.9']
title = "Email parser creates a message object that can't be flattened"
updated_at = <Date 2020-10-19.23:11:42.432>
user = 'https://github.com/msapiro'

bugs.python.org fields:

activity = <Date 2020-10-19.23:11:42.432>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-19.23:02:01.155>
closer = 'barry'
components = ['email']
creation = <Date 2016-06-14.18:52:52.474>
creator = 'msapiro'
dependencies = []
files = ['43391', '43394']
hgrepos = []
issue_num = 27321
keywords = ['patch']
message_count = 22.0
messages = ['268580', '268589', '279172', '295080', '295093', '295101', '295105', '295107', '295108', '295430', '295442', '295443', '296293', '310513', '311272', '327136', '336930', '378083', '378088', '379052', '379056', '379057']
nosy_count = 10.0
nosy_names = ['barry', 'msapiro', 'diekhans', 'r.david.murray', 'maciej.szulik', 'matrixise', 'cheryl.sabella', 'Johannes L\xc3\xb6thberg', 'miss-islington', 'edeca']
pr_nums = ['1977', '18074', '22796', '22797']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27321'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[msapiro]

The attached file, bad_email, can be parsed via

msg = email.message_from_binary_file(open('bad_email', 'rb'))

but then msg.as_string() prodices the following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/email/message.py", line 159, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 189, in _write
    msg.replace_header('content-transfer-encoding', munge_cte[0])
  File "/usr/lib/python3.5/email/message.py", line 559, in replace_header
    raise KeyError(_name)
KeyError: 'content-transfer-encoding'

[msapiro]

One additional observation. The original message contained no Content-Transfer-Encoding header even though the message body was raw koi8-r characters. Adding

Content-Transfer-Encoding: 8bit

to the message headers avoids the issue, but that is not a practical solution as the message was Russian spam received by a Mailman list and the resultant KeyError caused problems in Mailman.

We can work on defending against this in Mailman, but I suggest that the munge_cte code in generator._write() avoid the documented possible KeyError raised by replace_header() by using __delitem__() and __setitem__() instead as in the attached generator.patch.

[warsaw]

While I was reviewing https://gitlab.com/mailman/mailman/merge_requests/197/diffs I noticed the KeyError and it made me thing "hmm, I wonder if this should be turned into one of the email package errors"?

[kyrias]

Any updates on this? I'm having the same problem with some non-spam emails while trying to use some mail-handling tools written in Python.

[bitdancer]

replace_header has a different semantic than del-and-set (replace_header leaves the header in the same location in the list, rather than appending it to the end...that's it's purpose). If replace_header is throwing a key error, then I guess we need a look-before-you-leap if statement.

And a test :)

Note that a correct fix would preserve the broken input email, but since we're obviously already doing munging I'm fine with just making this work for now.

[msapiro]

I considered look before you leap, but I decided since we're munging the headers anyway, preserving their order is not that critical, but the patch is easy enough. I'll work on that and a test.

[kyrias]

Fix: kyrias@a986a82
Testcase: kyrias@9a51042

Can start a PR once my CLA signature goes through I guess.

[msapiro]

It looks like Johannes beat me to it. Thanks for that, but see my comments in the diff at kyrias@a986a82

[kyrias]

Ah, didn't even see your comment before I did it! Fix to the comments are on the same branch, will be rebased before PR is up.

[matrixise]

maybe we could merge the PR, and I could propose a backport for 3.5 and 3.6.

2.7 is affected ?

[bitdancer]

I'm going to try to review this this weekend.

[matrixise]

ok, I have tested on 3.6 and 3.5, just with the test. and in this case, we get the errors on both.
if we apply the patch of Johannes, the test passes and there is no issues.

+1 the backports for 3.5 and 3.6 is just a git cherry-picking.

[kyrias]

Ping?

[bitdancer]

Note: I reviewed this a while ago but the review comments haven't been addressed.

[bitdancer]

Requested a small additional change to the new tests, and then this will be ready to go in.

[edeca]

Ping on an ETA for this fix?

[csabella]

@r.david.murray, it appears that all your requested changes have been addressed on the PR. Please re-review this when you get a chance. Thanks!

[diekhans]

any chance of getting this merged? A work-around is not obvious

[msapiro]

I work around it with

class Message(email.message.Message):

    def as_string(self):
        # Work around for https://bugs.python.org/issue27321 and
        # https://bugs.python.org/issue32330.
        try:
            value = email.message.Message.as_string(self)
        except (KeyError, LookupError, UnicodeEncodeError):
            value = email.message.Message.as_bytes(self).decode(
                'ascii', 'replace')
        # Also ensure no unicode surrogates in the returned string.
        return email.utils._sanitize(value)

This is easy for me because it's Mailman which already subclasses email.message.Message for other reasons. It is perhaps more difficult if you aren't already subclassing email.message.Message for other purposes.

[miss-islington]

New changeset bf83822 by Mark Sapiro in branch 'master':
bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074)
bf83822

[miss-islington]

New changeset 371146a by Miss Skeleton (bot) in branch '3.8':
bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074)
371146a

[miss-islington]

New changeset 72ce82a by Miss Skeleton (bot) in branch '3.9':
bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074)
72ce82a