non-ascii characters in headers causes TypeError on email.policy.Policy.fold

[rad164]

BPO	33524
Nosy	@warsaw, @bitdancer, @matrixise, @csabella, @miss-islington
PRs	bpo-33524: Fix the behavior when max_line_length is 0 or None in email.policy #6966 bpo-33524: Fix the folding of email header when max_line_length is 0 or None #13391 [3.7] bpo-33524: Fix the folding of email header when max_line_length is 0 or None (GH-13391) #13393

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-17.20:54:43.655>
created_at = <Date 2018-05-15.16:35:30.339>
labels = ['3.8', 'type-bug', '3.7', 'expert-email']
title = 'non-ascii characters in headers causes TypeError on email.policy.Policy.fold'
updated_at = <Date 2019-05-17.20:54:43.654>
user = 'https://bugs.python.org/rad164'

bugs.python.org fields:

activity = <Date 2019-05-17.20:54:43.654>
actor = 'cheryl.sabella'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-17.20:54:43.655>
closer = 'cheryl.sabella'
components = ['email']
creation = <Date 2018-05-15.16:35:30.339>
creator = 'rad164'
dependencies = []
files = []
hgrepos = []
issue_num = 33524
keywords = ['patch']
message_count = 6.0
messages = ['316680', '316681', '335436', '342758', '342763', '342765']
nosy_count = 7.0
nosy_names = ['barry', 'r.david.murray', 'matrixise', 'altvod', 'cheryl.sabella', 'miss-islington', 'rad164']
pr_nums = ['6966', '13391', '13393']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33524'
versions = ['Python 3.7', 'Python 3.8']

[rad164]

Python 3.6.5 has trouble on folding email messages:

Traceback (most recent call last):
  File "emailtest.py", line 7, in <module>
    policy.fold("Subject", msg["Subject"])
  File "/usr/lib/python3.6/email/policy.py", line 183, in fold
    return self._fold(name, value, refold_binary=True)
  File "/usr/lib/python3.6/email/policy.py", line 205, in _fold
    return value.fold(policy=self)
  File "/usr/lib/python3.6/email/headerregistry.py", line 258, in fold
    return header.fold(policy=policy)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 144, in fold
    return _refold_parse_tree(self, policy=policy)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 2651, in _refold_parse_tree
    part.ew_combine_allowed, charset)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 2728, in _fold_as_ew
    first_part = to_encode[:text_space]
TypeError: slice indices must be integers or None or have an __index__ method

The message has non-ascii characters in header and set max_line_length=0, regardless length of the header. Here is the code to reproduce.

from email.message import EmailMessage
from email.policy import default

policy = default.clone(max_line_length=0)
msg = EmailMessage()
msg["Subject"] = "á"
policy.fold("Subject", msg["Subject"])

I first found this issue on Maildir.add, which saves the message to a file without word wrap.

[bitdancer]

Oh, interesting.

I could argue that that's a missing feature in Python's slice handling. The value of text_space at that point is '+inf', and I obviously incorrectly assumed that slicing would treat that as if it were [:]. The counter argument, of course, is that inf is a float not an integer. I suppose we'll have to use sys.maxsize instead.

[matrixise]

See this issue as a duplicata of this one:

https://bugs.python.org/issue35985

[bitdancer]

New changeset feac6cd by R. David Murray (Abhilash Raj) in branch 'master':
bpo-33524: Fix the folding of email header when max_line_length is 0 or None (bpo-13391)
feac6cd

[miss-islington]

New changeset 5386aaf by Miss Islington (bot) in branch '3.7':
bpo-33524: Fix the folding of email header when max_line_length is 0 or None (GH-13391)
5386aaf

[csabella]

Thank you, @Licht-T for the original patch and @maxking for the rebase. Also, thank you, @r.david.murray for the review and merge.