AttributeError when parsing multipart email with invalid non-decodable Content-Transfer-Encoding

[ajdlinux]

BPO	30835
Nosy	@warsaw, @bitdancer, @ajdlinux, @miss-islington, @websurfer5
PRs	bpo-30835: email: Fix AttributeError when parsing invalid Content-Tra… #2544 bpo-30835: email: Fix AttributeError when parsing invalid CTE #13598 [3.7] bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598) #13820 [3.6] bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598) #13821
Files	testprog.py: Example program that demonstrates the error

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-07-03.13:37:03.005>
labels = ['3.7', '3.8', 'type-bug', 'library', 'expert-email']
title = 'AttributeError when parsing multipart email with invalid non-decodable Content-Transfer-Encoding'
updated_at = <Date 2019-06-05.10:23:41.770>
user = 'https://github.com/ajdlinux'

bugs.python.org fields:

activity = <Date 2019-06-05.10:23:41.770>
actor = 'miss-islington'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2017-07-03.13:37:03.005>
creator = 'Andrew Donnellan'
dependencies = []
files = ['46989']
hgrepos = []
issue_num = 30835
keywords = ['patch']
message_count = 4.0
messages = ['297584', '297771', '344620', '344698']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'Andrew Donnellan', 'miss-islington', 'Jeffrey.Kintscher']
pr_nums = ['2544', '13598', '13820', '13821']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30835'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[ajdlinux]

Parsing an email containing a multipart Content-Type, along with a Content-Transfer-Encoding containing an invalid (non-ASCII-decodable) byte will fail.

email.feedparser.FeedParser._parsegen() calls "self._cur.get('content-transfer-encoding', '8bit')" to get the header.

It then tries to check whether the C-T-E is in the allowable set of ('7bit', '8bit', 'binary'), and to do so case-insensitively, it tries to convert the header to lowercase. However, because there's an invalid character in there, it's dealing with a Header object rather than a str. Hence it throws an AttributeError.

Correct behaviour would be to convert the Header to a str, see that it's not valid, and continue on to handle the defect as usual.

Thanks to Daniel Axtens for finding this bug as he was running the AFL fuzzer on the email parsing code in Patchwork (https://github.com/getpatchwork/patchwork).

Pull request incoming.

[bitdancer]

There's a deeper problem here involving how Header is used in compat32 that I've been aware of for a while but haven't had time to try to think through a fix for (there may not be one, given the history of the compat32 code). In the meantime, the proposed fix is reasonable. (It isn't needed for the new policies, but it doesn't hurt.)

[warsaw]

New changeset aa79707 by Barry Warsaw (Abhilash Raj) in branch 'master':
bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598)
aa79707

[miss-islington]

New changeset f62a372 by Miss Islington (bot) in branch '3.7':
bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598)
f62a372

[furkanonder]

@warsaw The issue seems to be solved. Can we close it?