smtplib: verify breaks with Postfix servers

[mpg]

BPO	7484
Nosy	@bitdancer, @felipecruz
Files	bsl.py: demonstration issue7484-trunk.diff: trunk patch issue7484-py3k.diff: py3k patch issue7484-py3k.diff: py3k patch issue7484-27.diff issue7484-py3k_2.diff: improved patch issue7484-27_2.diff: improved patch 88b5c7ab7a03.diff unnamed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-07-19.02:09:54.970>
created_at = <Date 2009-12-12.06:55:52.562>
labels = ['easy', 'type-bug', 'library']
title = 'smtplib: verify breaks with Postfix servers'
updated_at = <Date 2011-07-19.02:21:24.877>
user = 'https://bugs.python.org/mpg'

bugs.python.org fields:

activity = <Date 2011-07-19.02:21:24.877>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2011-07-19.02:09:54.970>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2009-12-12.06:55:52.562>
creator = 'mpg'
dependencies = []
files = ['15530', '15674', '15675', '21641', '21642', '21651', '21652', '22680', '22693']
hgrepos = ['43']
issue_num = 7484
keywords = ['patch', 'easy']
message_count = 13.0
messages = ['96284', '96298', '96884', '133629', '133637', '133684', '140260', '140527', '140643', '140644', '140645', '140647', '140648']
nosy_count = 6.0
nosy_names = ['r.david.murray', 'pablomouzo', 'mpg', 'catalin.iacob', 'python-dev', 'felipecruz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7484'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[mpg]

Hi,

The verify method of SMTP objects created with smtplib doesn't work
properly with servers running Postfix, due to quoting problems: the
address is enclosed in pointed brackets by the method, which changes the
way it is interpreted by the server.

The attached demo file uses the mx1.nic.fr server, which runs Postfix
and exhibits the problem at the time of the writing.

RFC 5321 says the argument of VRFY is a string representing a "user
name", without saying much about what a "user name" is, but nothing
suggests it should be quoted in pointed brackets. Moreover, the example
in D.4 doens't use any quoting.

Anyway, even if Postfix was wrong, I think it would be worth trying to
support it, since it is quite widely used.

Thanks!

[bitdancer]

I agree. My reading of the rfc is that the form without the brackets
*must* be supported by the MTA, while any other form is optional. So
smtplib should use the required form for its VRFY query. Any MTA that
doesn't recognize that form would be broken.

2.5 is in security-fix-only mode, so this can only be fixed in 2.6 and
above.

[pablomouzo]

This patch solves the problem with the VRFY command, but I'm still
wondering if this happens with other commands too.

[felipecruz]

I've rewrote those patches to 'default' and 2.7

[bitdancer]

Thanks for working on this.

The tests seem to be missing, as is the line that adds 'clean' to the def, so the patches won't work as is.

However, now that I've looked at the patch in more detail, adding a parameter to a public method is not something we can do in a bug fix release. So, this solution would work for 3.3, but not for 2.7 and 3.2. In any case, Guido thinks that parameters that have only two values should be replaced by methods with two different names. In this case that makes a lot of sense. I've checked the RFC and the code, and there are two cases: MAIL FROM and RCPT TO, which require the address to be in <>s, and VRFY and EXPN, which prefer that it not be in <>s. So I think we should introduce a new, private function for use in the VRFY and EXPN cases:

    def _addronly(addr):
        (fullname, email) = email.utils.parseaddr(addr)
        return email

Can you do a new patch, adding the above function and using it at the right places? Tests are also needed...it should be possible to modify the test that the original patch modified so that it checks to make sure the <> are not added. If you need help with that let me know.

[felipecruz]

David..

I extracted quoteaddr code to _addrformat and now quoteaddr and _addronly call _addrformat passing a format (<%s> or %s).

I've also created quoteaddr and _addronly test functions as well modified VRFY and EXPN tests to make sure they call _addronly and pointed brackets aren't added.

Let me know if those patches still need improvements.

[felipecruz]

Can anyone take a loot at those patches?

Do they need more tests?

[cataliniacob]

I looked at the Felipe's patch and hopefully made some improvements.

Unlike Felipe's patch I didn't change the reply of the SMTP server in the tests but instead use what VRFY and EXPN actually send to index the users and lists dictionaries. If <> would be sent the lookup would fail. Similarly, when VRFY return 550 it echoed the address as received and now it's tested to be equal to something without <>.

By the way, but I was wondering:

• is the try/except really needed or just a historical artifact (why would email.utils.parseaddr raise AttributeError?)
• is the test to None correct? It was added by the fix to bpo-1430298 but does email.utils.parseaddr ever return None for the address? (I could only get it to return '')
  I kept quoteaddr as is to make it easier to review the patch but if David confirms the above points are valid I can create new issues for them and simplify/fix quoteaddr.

[python-dev]

New changeset c4d884d5d86c by R David Murray in branch '2.7':
bpo-7484: no more <> around addresses in VRFY or EXPN
http://hg.python.org/cpython/rev/c4d884d5d86c

New changeset f8c4ac9aa9e2 by R David Murray in branch '3.2':
bpo-7484: no more <> around addresses in VRFY or EXPN
http://hg.python.org/cpython/rev/f8c4ac9aa9e2

New changeset 0d9216de8f05 by R David Murray in branch 'default':
Merge bpo-7484: no more <> around addresses in VRFY or EXPN
http://hg.python.org/cpython/rev/0d9216de8f05

[python-dev]

New changeset 50b6c3053c30 by R David Murray in branch 'default':
bpo-7484: simplify quoteaddr: if parseaddr throws an error it is a bug.
http://hg.python.org/cpython/rev/50b6c3053c30

[bitdancer]

Thank you both for your work on this. The patch I committed is a combination of my _addr_only, Filipe's tests, and Catalin's modifications to those tests. quoteaddr, although in the __all__, is not documented and is really an implementation detail, as is the new _addr_only. So I am only testing them indirectly through the documented parts of the API (I added a test for <> address, and one for an IDNA encoded address).

Catalin, I think you are correct about the try/except/None stuff. As far as I can tell it is left over from the old days before the email package and its philosophy of never throwing parsing errors. Nowadays if parseaddr throws an error, it is a bug. That's a refactoring not a bug fix, though, so I didn't backport it.

[felipecruz]

You're very kind David.

Hope I can contribute with something more relevant next time :)

best regards,
Felipe

2011/7/18 R. David Murray <report@bugs.python.org>

R. David Murray <rdmurray@bitdance.com> added the comment:

Thank you both for your work on this. The patch I committed is a
combination of my _addr_only, Filipe's tests, and Catalin's modifications to
those tests. quoteaddr, although in the __all__, is not documented and is
really an implementation detail, as is the new _addr_only. So I am only
testing them indirectly through the documented parts of the API (I added a
test for <> address, and one for an IDNA encoded address).

Catalin, I think you are correct about the try/except/None stuff. As far
as I can tell it is left over from the old days before the email package and
its philosophy of never throwing parsing errors. Nowadays if parseaddr
throws an error, it is a bug. That's a refactoring not a bug fix, though,
so I didn't backport it.

----------
resolution: -> fixed
stage: test needed -> committed/rejected
status: open -> closed

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue7484\>

---

[bitdancer]

Don't short change yourself. This bug would still be open if it hadn't been for your work, regardless of how much of it wound up in the final patch :)