email.header error during .flatten()

[sdaoden]

BPO	11401
Nosy	@bitdancer
Files	issue_11401_mail.txt issue_11401_runner.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/bitdancer'
closed_at = <Date 2011-03-16.22:30:15.162>
created_at = <Date 2011-03-04.20:49:44.635>
labels = ['type-bug', 'library']
title = 'email.header error during .flatten()'
updated_at = <Date 2011-03-16.22:30:15.160>
user = 'https://bugs.python.org/sdaoden'

bugs.python.org fields:

activity = <Date 2011-03-16.22:30:15.160>
actor = 'r.david.murray'
assignee = 'r.david.murray'
closed = True
closed_date = <Date 2011-03-16.22:30:15.162>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2011-03-04.20:49:44.635>
creator = 'sdaoden'
dependencies = []
files = ['21005', '21006']
hgrepos = []
issue_num = 11401
keywords = []
message_count = 8.0
messages = ['130072', '130118', '130259', '130264', '131058', '131103', '131187', '131188']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'sdaoden', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11401'
versions = ['Python 3.1', 'Python 3.2', 'Python 3.3']

[sdaoden]

Hello, David, the error from bpo-11116 occurred again,
and it still can be healed with the patch file20675.
(I'm opening a new issue because 11116 turned over to
the mailbox fix.
My repo and installation is at:

21:34 ~/arena/code.extern.repos/py3k.hg $ hg identify
ddd8bf84e27d+ (py3k) tip
21:34 ~/arena/code.extern.repos/py3k.hg $ hg slog -r tip
changeset: 10028:ddd8bf84e27d
branch: py3k
tag: tip
user: eli.bendersky
date: Fri Mar 04 11:38:14 2011 +0100
summary: [svn r88742]

Here is the dump:

| Opening mbox mailbox /private/var/folders/Da/DaZX3-k5G8a57zw6MSmjJ++++TM/-Tmp-/mail/test
David: <class 'mailbox.mbox'>  <->  <class 'mailbox.MaildirMessage'>
PANIC: [box] test: message-add failed.
  Traceback (most recent call last):
    File "/Users/steffen/usr/bin/s-postman.py", line 1093, in save_ticket
      mb.add(ticket.message())
	...
        try:
            print('David:', type(mb), ' <-> ', type(ticket.message()))
            mb.add(ticket.message())
            mb.flush()
        except Exception as e:
            efun('[box] ', self.id, ': message-add failed.\n', extb=E())
        finally:
            mb.unlock()
	...
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 595, in add
      self._toc[self._next_key] = self._append_message(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 733, in _append_message
      offsets = self._install_message(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 805, in _install_message
      self._dump_message(message, self._file, self._mangle_from_)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 215, in _dump_message
      gen.flatten(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 88, in flatten
      self._write(msg)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 141, in _write
      self._write_headers(msg)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 373, in _write_headers
      self.write(header.encode(linesep=self._NL)+self._NL)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/header.py", line 317, in encode
      formatter.feed(lines[0], charset)
  Exception: IndexError: list index out of range

In the meanwhile the thing has dispatched tens of thousands
of mails, and no error did ever occur (reading from MBox,
storing as Maildir). This happened when saving the first
message read from a Maildir mailbox to a MBox one.
I think i'll gonna do some round-trip tests tomorrow and
monday evening+. Anyway, patching in file20675 heals the
problem and the thing finishs gracefully.

[sdaoden]

Changes by R. David Murray <rdmurray@bitdance.com>:
assignee: -> r.david.murray

David, i'm sending you data to reproduce the error on your local
machine!

Please create a test directory and place the files therein
(an output.mbox will be created). Just do

mkdir -p issue_11401/input.mdir/{cur,new,tmp}
mv issue_11401_mail.txt "issue_11401/input.mdir/cur/test:2,S"
mv issue_11401_runner.py issue_11401/
cd issue_11401
python3 issue_11401_runner.py

(I'm looking forward for a glass of red wine, just in case we'll
ever meet.-)

[sdaoden]

David, it seems my patch http://bugs.python.org/file20675/email_header.patch
is a real requirement for EMAIL 5.1 code.

This is because BytesGenerator._write_headers() creates a Header
instance in 'else:' (the other branches are not entered at all for
the test mail), passing 'v' as the 's=' arg of Header ctor.
It follows that Header._chunks[] is never the empty list.
'v' however may be the empty string for header fields like
'X-Status:', 'X-Keywords:' and maybe more, resulting in
string.splitlines() trying to break up the empty string.

I don't understand why i could pass thousands of mails, including
the test mail, dozens of times through my thing without showing
this up? Does the mailbox.mbox message subclass adjust such things?
Bye.

[bitdancer]

I plan to take a look at this and other email bugs during the Pycon sprints, if not before.

[bitdancer]

It turns out this is a bug in 3.1, not something introduced by email5.1 in 3.2. The minimum reproducer is stringifying any message containing a header with no body.

[sdaoden]

On Tue, Mar 15, 2011 at 10:44:43PM +0000, R. David Murray wrote:

header with no body.

(Those are the *real* dangerous ones!)

[python-dev]

New changeset d9c3cfd36b58 by R David Murray in branch '3.1':
bpo-11401: handle headers with no value.
http://hg.python.org/cpython/rev/d9c3cfd36b58

New changeset 2de45ced8291 by R David Murray in branch '3.2':
Merge bpo-11401 fix from 3.1.
http://hg.python.org/cpython/rev/2de45ced8291

New changeset e318d84efe6e by R David Murray in branch 'default':
Merge bpo-11401 fix from 3.2.
http://hg.python.org/cpython/rev/e318d84efe6e

[bitdancer]

OK, now this bit works like it did in Python2.