bpo-45001: Make email date parsing more robust against malformed input #27946

wbolster · 2021-08-25T13:27:07Z

Various date parsing utilities in the email module, such as
email.utils.parsedate(), are supposed to gracefully handle invalid
input, typically by raising an appropriate exception or by returning
None.

The internal email._parseaddr._parsedate_tz() helper used by some of
these date parsing routines tries to be robust against malformed input,
but unfortunately it can still crash ungracefully when a non-empty but
whitespace-only input is passed. This manifests as an unexpected
IndexError.

In practice, this can happen when parsing an email with only a newline
inside a ‘Date:’ header, which unfortunately happens occasionally in the
real world.

Here's a minimal example:

$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.utils
>>> email.utils.parsedate('foo')
>>> email.utils.parsedate(' ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate
    t = parsedate_tz(data)
  File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz
    if data[0].endswith(',') or data[0].lower() in _daynames:
IndexError: list index out of range

The fix is rather straight-forward: guard against empty lists, after
splitting on whitespace, but before accessing the first element.

https://bugs.python.org/issue45001

See https://bugs.python.org/issue45001 Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element.

ambv · 2021-08-26T14:23:36Z

Also added backport to 3.8 since passing invalid headers via e-mail can lead to DoS here.

wbolster · 2021-08-26T14:33:58Z

Also added backport to 3.8 since passing invalid headers via e-mail can lead to DoS here.

a denial of service is exactly what can happen indeed. practically speaking, the widely used imapclient indirectly calls this code, and as a result it unexpectedly crashes when listing mailbox contents if such a message is encountered.

miss-islington · 2021-08-26T14:49:05Z

Thanks @wbolster for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.6, 3.7, 3.8, 3.9, 3.10.
🐍🍒⛏🤖

bedevere-bot · 2021-08-26T14:49:17Z

GH-27972 is a backport of this pull request to the 3.10 branch.

bedevere-bot · 2021-08-26T14:49:21Z

GH-27973 is a backport of this pull request to the 3.9 branch.

pythonGH-27946) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

bedevere-bot · 2021-08-26T14:49:26Z

GH-27974 is a backport of this pull request to the 3.8 branch.

bedevere-bot · 2021-08-26T14:49:31Z

GH-27975 is a backport of this pull request to the 3.7 branch.

bedevere-bot · 2021-08-26T14:49:36Z

GH-27976 is a backport of this pull request to the 3.6 branch.

pythonGH-27946) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

GH-27946) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

GH-27946) (GH-27973) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

GH-27946) (GH-27974) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

GH-27946) (GH-27975) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

GH-27946) (GH-27976) Various date parsing utilities in the email module, such as email.utils.parsedate(), are supposed to gracefully handle invalid input, typically by raising an appropriate exception or by returning None. The internal email._parseaddr._parsedate_tz() helper used by some of these date parsing routines tries to be robust against malformed input, but unfortunately it can still crash ungracefully when a non-empty but whitespace-only input is passed. This manifests as an unexpected IndexError. In practice, this can happen when parsing an email with only a newline inside a ‘Date:’ header, which unfortunately happens occasionally in the real world. Here's a minimal example: $ python Python 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.utils >>> email.utils.parsedate('foo') >>> email.utils.parsedate(' ') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate t = parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz res = _parsedate_tz(data) File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz if data[0].endswith(',') or data[0].lower() in _daynames: IndexError: list index out of range The fix is rather straight-forward: guard against empty lists, after splitting on whitespace, but before accessing the first element. (cherry picked from commit 989f6a3) Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

wbolster requested a review from a team as a code owner August 25, 2021 13:27

the-knights-who-say-ni added the CLA signed label Aug 25, 2021

bedevere-bot added the awaiting review label Aug 25, 2021

wbolster force-pushed the bpo45001-date-email-parsing-robustness branch from 0edbd8d to d75760c Compare August 25, 2021 14:34

wbolster changed the title ~~bpo45001: Make email date parsing more robust against malformed input~~ bpo-45001: Make email date parsing more robust against malformed input Aug 25, 2021

ambv added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes needs backport to 3.8 only security fixes labels Aug 26, 2021

Add Blurb

d8d700d

ambv added needs backport to 3.7 labels Aug 26, 2021

ambv merged commit 989f6a3 into python:main Aug 26, 2021

bedevere-bot removed awaiting review needs backport to 3.10 only security fixes labels Aug 26, 2021

bedevere-bot removed the needs backport to 3.9 only security fixes label Aug 26, 2021

bedevere-bot removed the needs backport to 3.8 only security fixes label Aug 26, 2021

bedevere-bot removed the needs backport to 3.7 label Aug 26, 2021

bedevere-bot removed the needs backport to 3.6 label Aug 26, 2021

wbolster deleted the bpo45001-date-email-parsing-robustness branch August 26, 2021 15:03

wbolster mentioned this pull request Aug 27, 2021

[3.7] bpo-45001: Make email date parsing more robust against malformed input (GH-27946) #27975

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-45001: Make email date parsing more robust against malformed input #27946

bpo-45001: Make email date parsing more robust against malformed input #27946

wbolster commented Aug 25, 2021 •

edited by bedevere-bot

Loading

ambv commented Aug 26, 2021

wbolster commented Aug 26, 2021

miss-islington commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bpo-45001: Make email date parsing more robust against malformed input #27946

bpo-45001: Make email date parsing more robust against malformed input #27946

Conversation

wbolster commented Aug 25, 2021 • edited by bedevere-bot Loading

ambv commented Aug 26, 2021

wbolster commented Aug 26, 2021

miss-islington commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

bedevere-bot commented Aug 26, 2021

wbolster commented Aug 25, 2021 •

edited by bedevere-bot

Loading