getting HTTP 400s sending emails with sender that contains extra chars

[melgart]

hi there:

before finding AnyMail (which is awesome), we used to use the plain ol SMTP-based backends and when we send outbound emails with this kind of sender_email (below), things just worked:

'Foobar, inc. (via SomeOtherFoobar)" do-not-reply@foobar.com'

after switching to AnyMail using MailGun as the backend, now i started seeing HTTP 400 errors claiming that my 'from' address was not valid:

ESP API response 400:
{
  "message": "'from' parameter is not a valid address. please check documentation"
}

The thing is, if i use curl and POST new messages to Mailgun HTTP APIs directly, that sender_email is accepted. i used a curl statement like this:

curl -s --user 'api:<secret!>' \
    https://api.mailgun.net/v3/<mydomain>/messages \
    -F from='Foobar Construction & Design, LLC. (via FoobarsRUs) <do-not-reply@foobar.com>' \
    -F to=test@test.com \
    -F subject='Hello' \
    -F text='Testing some Mailgun awesomness!'

so after lots of testing, i found that i needed to double-quote the part of the sender_email that might contain commas, parentheses, etc: "Foobar Construction & Design, LLC. (via FoobarsRUs)"

doing this, then AnyMail sends the message properly. it just seems odd that other email backends worked for this sender, as does the MailGun HTTP APIs directly.

i added a test script to repro this issue, you might need to edit the To email address of course. thank you for any help!

matias elgart

test_anymail.py.txt

[medmunds]

In your test script, it looks like you're missing an opening double-quote on the "realname" part of the email address:

    sender = 'Foobar, inc. (via SomeOtherFoobar)" <do-not-reply@foobar.com>'
    #         ^ missing opening " to match here ^

(Your curl example doesn't have any double quotes, so it's not testing the same case.)

Anymail should accept -- and pass correctly to your ESP -- any validly-formatted email addresses. BTW, the double quotes shouldn't be necessary. Django's EmailMessage API makes you split multiple emails into a python list, so there's no confusion over commas: they're always assumed to be part of the "realname":

    # Either of these should work: matching double-quotes on realname:
    sender = '"Foobar, inc. (via SomeOtherFoobar)" <do-not-reply@foobar.com>'
    # or no quotes on realname:
    sender = 'Foobar, inc. (via SomeOtherFoobar) <do-not-reply@foobar.com>'

[Edit: above comment is wrong -- the double quotes are required for a validly-formatted email address]

[melgart]

very sorry about that -- typo while setting up my test file; i used the same test file to break it first, then fix it by surrounding the 'real name' part with double quotes, as you mentioned.

removing that stray double quote so that the sender email is:

'Foobar, inc. (via SomeOtherFoobar) do-not-reply@foobar.com'

still fails when sending to Mailgun though, but the issue is with the python email package being used in django-anymail.anymail.utils.py, specifically the email.parseaddr function used in the ParsedEmail class. here's a quick test to show how email.parseaddr() deals with an email address like that:

from email.utils import parseaddr
email1 = 'Foobar, inc. (via SomeOtherFoobar) <do-not-reply@foobar.com>'  # problem addr
print parseaddr(email1)
email2 = '"Foobar, inc. (via SomeOtherFoobar)" <do-not-reply@foobar.com>'
print parseaddr(email2)

which returns:

(pw_ci)wintermute:app 21:23:38 melgart$ python test_email_parseaddr.py
('', 'Foobar')  # <-- unexpected
('Foobar, inc. (via SomeOtherFoobar)', 'do-not-reply@foobar.com')  # expected

i just wanted to pass along this info in case any of your other users ran into this problem. the fix from the sender's side is trivial, just not documented and it's a little confusing because Mailgun's email validation endpoint (https://api.mailgun.net/v3/address/validate) says this email address (above) is NOT valid, even with the double quotes around the real name part, yet the curl command below works, even without the real name in double quotes:

curl -s --user 'api:<secret>' \
    https://api.mailgun.net/v3/<mydomain>/messages \
    -F from='Foobar Construction & Design, LLC. (via FoobarsRUs) <do-not-reply@foobar.com>' \
    -F to=test@test.com \
    -F subject='Hello' \
    -F text='Testing some Mailgun awesomness!'

so, i'm thoroughly confused but i understand how to proceed. thanks for the info and again, thank you for your work on AnyMail -- a very nice package to use.

[medmunds]

Ah, thanks for tracking it down to the python library email.utils.parseaddr call.

Parsing email addresses -- even without the realname part -- is notoriously complex. (Which is why ordinary programmers like me are supposed to delegate the work to well-tested library calls.)

There's definitely a bug here in Anymail: parseaddr actually parses an address list and returns the (realname, email) of the first address in the list, if any. The python docs are a little ambiguous on this. (Also, good find! This has been broken for years, way back into Anymail's predecessor package Djrill.)

From muddling through RFC 5322, I'm pretty sure the double quotes are required for a valid email address that does what you intend. (Omitting the quotes from a "display-name" is allowed as an "obs-phrase". But an obs-phrase can't contain a comma.) I'd guess Mailgun has relaxed validation for the address fields in its API.

The python library getaddresses call correctly interprets the problem address without the quotes as two separate addresses:

>>> from email.utils import getaddresses
>>> email1 = 'Foobar, inc. (via SomeOtherFoobar) <do-not-reply@foobar.com>'  # problem addr
>>> getaddresses([email1])
[('', 'Foobar'), ('inc. (via SomeOtherFoobar)', 'do-not-reply@foobar.com')]

Anymail needs to switch from parseaddr to getaddresses, and needs to raise an error when what should be a single email address actually contains multiple ones.