IndexError thrown on email.message.Message.get

[uckelman]

BPO	29412
Nosy	@warsaw, @bitdancer, @maxking, @zhangyangyu, @uckelman
PRs	bpo-29412: patched string index out of range error in get_word function of _head… #6907 bpo-29412: Fix indexError when parsing a header value ending unexpectedly #14387 [3.8] bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) #14411 [3.7] bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) #14412
Files	29412.patch: patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-17.03:01:56.392>
created_at = <Date 2017-02-01.13:53:21.553>
labels = ['3.8', 'type-bug', '3.7', 'expert-email', '3.9']
title = 'IndexError thrown on email.message.Message.get'
updated_at = <Date 2019-08-17.03:01:56.384>
user = 'https://github.com/uckelman'

bugs.python.org fields:

activity = <Date 2019-08-17.03:01:56.384>
actor = 'maxking'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-17.03:01:56.392>
closer = 'maxking'
components = ['email']
creation = <Date 2017-02-01.13:53:21.553>
creator = 'uckelman'
dependencies = []
files = ['46562']
hgrepos = []
issue_num = 29412
keywords = ['patch']
message_count = 13.0
messages = ['286631', '286633', '286635', '286845', '286867', '287232', '287242', '345631', '345632', '345633', '346676', '346689', '346690']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'maxking', 'xiang.zhang', 'uckelman']
pr_nums = ['6907', '14387', '14411', '14412']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29412'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[uckelman]

Test case:

  import email
  import email.policy
 
  txt = '''From: juckelman@strozfriedberg.co.uk
  To: (Recipient list suppressed)
  Date: Thu, 22 Aug 2013 04:13:02 +0000
  Subject: ADSF-1082
 
  Hey!
  '''
 
  msg = email.message_from_string(txt)
  msg.get('to')
  msg = email.message_from_string(txt, policy=email.policy.default)
  msg.get('to')

The second msg.get() throws an IndexError:

  Traceback (most recent call last):
    File "test.py", line 16, in <module>
      print(msg.get('to'))    # throws IndexError
    File "/usr/lib64/python3.5/email/message.py", line 472, in get
      return self.policy.header_fetch_parse(k, v)
    File "/usr/lib64/python3.5/email/policy.py", line 153, in header_fetch_parse
      return self.header_factory(name, ''.join(value.splitlines()))
    File "/usr/lib64/python3.5/email/headerregistry.py", line 586, in __call__
      return self[name](name, value)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 197, in __new__
      cls.parse(value, kwds)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 337, in parse
      kwds['parse_tree'] = address_list = cls.value_parser(value)
    File "/usr/lib64/python3.5/email/headerregistry.py", line 328, in value_parser
      address_list, value = parser.get_address_list(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2336, in get_address_list
      token, value = get_address(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2313, in get_address
      token, value = get_group(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2269, in get_group
      token, value = get_display_name(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 2095, in get_display_name
      token, value = get_phrase(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 1770, in get_phrase
      token, value = get_word(value)
    File "/usr/lib64/python3.5/email/_header_value_parser.py", line 1745, in get_word
      if value[0]=='"':
  IndexError: string index out of range

The docs say that email.policy.default has raise_on_defect set to False, hence parse errors ought to be reported via EmailMessage.defects, not by throwing an exception.

[bitdancer]

Does the patch from bpo-27931 fix your problem as well? I haven't looked closely enough to see if I think it should, I'm just hoping :)

[uckelman]

No dice. I get the same exception with issue27931_v2.patch. I briefly looked at the other two, and don't expect those will help, either.

[zhangyangyu]

This seems not related to bpo-27931.

The problem is that the receiver's content is only CFWS. It's just like it's empty and for the default policy, it checks value[0] == '"' in get_word.

[bitdancer]

I'm really short on time to even review patches these days, but I'll see if I can pry any loose if someone wants to propose a patch.

[uckelman]

I'm working on a patch now.

[uckelman]

Here's a patch, complete with tests.

If the value is all CFWS, then get_cfws(value)[1], which is what's left after the CFWS is extracted, is the empty string---which is why value[0] throws in this case.

[maxking]

I can't reproduce this problem with the latest master branch, it was perhaps fixed with some other PR.

This is also a dupe of bpo-31445.

@barry, @david: I think this issue can be closed.

[maxking]

For the record, this is how I tested using the master branch:

>>> msg = email.message_from_string('  To: (Recipient list suppressed)')
>>> msg['To']
>>> import email.policy
>>> msg = email.message_from_string('  To: (Recipient list suppressed)', policy=email.policy.default)
>>> msg
<email.message.EmailMessage object at 0x7f377512b370>
>>> msg['To']
>>> msg.get('to')

[maxking]

Nevermind, I was wrong, I was able to reproduce it:

>>> msg = email.message_from_string('To: (Recipient list suppressed)', policy=email.policy.default))
  File "<stdin>", line 1
SyntaxError: unmatched ')'
>>> msg = email.message_from_string('To: (Recipient list suppressed)', policy=email.policy.default)
>>> msg.get('to')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/maxking/Documents/cpython/Lib/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/home/maxking/Documents/cpython/Lib/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 589, in __call__
    return self[name](name, value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 197, in __new__
    cls.parse(value, kwds)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 340, in parse
    kwds['parse_tree'] = address_list = cls.value_parser(value)
  File "/home/maxking/Documents/cpython/Lib/email/headerregistry.py", line 331, in value_parser
    address_list, value = parser.get_address_list(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1951, in get_address_list
    token, value = get_address(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1928, in get_address
    token, value = get_group(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1884, in get_group
    token, value = get_display_name(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1710, in get_display_name
    token, value = get_phrase(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1385, in get_phrase
    token, value = get_word(value)
  File "/home/maxking/Documents/cpython/Lib/email/_header_value_parser.py", line 1360, in get_word
    if value[0]=='"':
IndexError: string index out of range

[warsaw]

New changeset 7213df7 by Barry Warsaw (Abhilash Raj) in branch 'master':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387)
7213df7

[warsaw]

New changeset b950cdb by Barry Warsaw (Miss Islington (bot)) in branch '3.7':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) (GH-14412)
b950cdb

[warsaw]

New changeset 82654a0 by Barry Warsaw (Miss Islington (bot)) in branch '3.8':
bpo-29412: Fix indexError when parsing a header value ending unexpectedly (GH-14387) (GH-14411)
82654a0