Email address display name fails with both encoded words and special chars

[bsiem]

BPO	37482
Nosy	@warsaw, @ned-deily, @bitdancer, @maxking, @miss-islington, @bsiem
PRs	bpo-37482: Fix email address name with encoded words and special chars #14561 [3.7] bpo-37482: Fix email address name with encoded words and special chars (GH-14561) #15371 [3.8] bpo-37482: Fix email address name with encoded words and specia… #15380
Files	email_header_test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-29.06:56:13.235>
created_at = <Date 2019-07-02.11:51:15.510>
labels = ['3.8', 'type-bug', '3.7', 'expert-email', '3.9']
title = 'Email address display name fails with both encoded words and special chars'
updated_at = <Date 2019-08-29.06:56:13.232>
user = 'https://github.com/bsiem'

bugs.python.org fields:

activity = <Date 2019-08-29.06:56:13.232>
actor = 'bsiem'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-29.06:56:13.235>
closer = 'bsiem'
components = ['email']
creation = <Date 2019-07-02.11:51:15.510>
creator = 'bsiem'
dependencies = []
files = ['48452']
hgrepos = []
issue_num = 37482
keywords = ['patch']
message_count = 10.0
messages = ['347136', '347628', '347634', '347637', '347925', '350128', '350130', '350707', '350711', '350736']
nosy_count = 6.0
nosy_names = ['barry', 'ned.deily', 'r.david.murray', 'maxking', 'miss-islington', 'bsiem']
pr_nums = ['14561', '15371', '15380']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37482'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[bsiem]

Special characters in email headers are normally put within double quotes. However, encoded words (=?charset?x?...?=) are not allowed withing double quotes. When the header contains a word with special characters and another word that must be encoded, the first one must also be encoded.

In the next example, The From header is quoted and therefore the comma is allowed; in the To header, the comma is not within quotes and not encoded, which is not allowed and rejected.

From: "Foo Bar, France" <foo@example.com>
To: Foo Bar, =?utf-8?q?Espa=C3=B1a?= <foo@example.com>

[bitdancer]

FYI, it would have been most helpful if you had posted your example in the issue text instead of as an attached file, as it explains the problem better than your text does :)

Here is a minimal reproducer:

>>> m = EmailMessage(policy=strict)
>>> m['From'] = '"Foo Bar, España" <foo@example.com>'
>>> bytes(m)
b'From: Foo Bar, =?utf-8?q?Espa=C3=B1a?= <foo@example.com>\n\n'

This serialization of the header is, as you say, invalid. Either the comma should be encoded, or the "Foo Bar," should be in quotes.

[bsiem]

Hello David, thank you for the suggestion.

Regarding your comment:

Either the comma should be encoded, or the "Foo Bar," should be in quotes.

According to RFC5322 the display name cannot contain both a quoted part and an encoded word, so the only option is to encode the comma.

Please let me know if I can do anything else.

[bitdancer]

The display name is a phrase, and a phrase is a sequence of words, and a word is either a quoted string or an atom. So it is legal to mix quoted strings and encoded words in a display name. I'd vote to do whichever one is easier to implement :) (I haven't looked at your PR yet and unfortunately my time is limited :(

[bsiem]

Yes, you are right! The fix is to encode the special characters.

[miss-islington]

New changeset df0c21f by Miss Islington (bot) (bsiem) in branch 'master':
bpo-37482: Fix email address name with encoded words and special chars (GH-14561)
df0c21f

[miss-islington]

New changeset c5bba85 by Miss Islington (bot) in branch '3.7':
bpo-37482: Fix email address name with encoded words and special chars (GH-14561)
c5bba85

[ned-deily]

New changeset bd127b1 by Ned Deily (bsiem) in branch '3.8':
[3.8] bpo-37482: Fix email address name with encoded words and special chars (GH-14561) (GH-15380)
bd127b1

[ned-deily]

I manually merged the stalled 3.8 backport to make 3.8.0b4. Can this issue now be closed?

[bsiem]

Thank you all!