email.Utils.py: UnicodeError in RFC2322 header

[qbin]

BPO	1414018
Nosy	@warsaw
Files	conversion_error.eml: sample email

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/warsaw'
closed_at = <Date 2006-07-28.03:19:44.000>
created_at = <Date 2006-01-24.20:19:55.000>
labels = ['library']
title = 'email.Utils.py: UnicodeError in RFC2322 header'
updated_at = <Date 2006-07-28.03:19:44.000>
user = 'https://bugs.python.org/qbin'

bugs.python.org fields:

activity = <Date 2006-07-28.03:19:44.000>
actor = 'barry'
assignee = 'barry'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-01-24.20:19:55.000>
creator = 'qbin'
dependencies = []
files = ['1883']
hgrepos = []
issue_num = 1414018
keywords = []
message_count = 2.0
messages = ['27352', '27353']
nosy_count = 2.0
nosy_names = ['barry', 'qbin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1414018'
versions = ['Python 2.4']

[qbin]

Description:
collapse_rfc2231_value does not handle UnicodeError
exception. Therefore a header like this one can cause
UnicodeError in attempting unicode conversion.

---
Content-Type: text/plain; charset="ISO-2022-JP"
Content-Disposition: attachment;
filename*=iso-2022-jp''%1B%24BJs9p%3Dq%2D%21%1B%28B%2Etxt
---

Test script:
---

#! /usr/bin/env python
import sys
import email

msg = email.message_from_file(sys.stdin)
for part in msg.walk():
  print part.get_params()
  print part.get_filename()

---

run
% env LANG=ja_JP.eucJP ./test.py < attached_sample.eml

Background:
Character 0x2d21 is invalid in JIS X0208 but defined in
CP932 (Shift_JIS's superset by Microsoft). Conversion
between Shift_JIS and ISO-2022-JP are computable
because both of them based on JIS X0208. So sometimes
CP932 characters appear in ISO-2022-JP encoded string,
typically produced by Windows MUA.
But Python's "ISO-2022-JP" means *pure* JIS X0208, thus
conversion is failed.

Workaround:
Convert to fallback_charset and/or skip invalid character.

[warsaw]

Logged In: YES
user_id=12800

r50894 for Python 2.4/email 3.0. This is already fixed in
Python 2.5/email 4.0