smtpd strips final carraige return from received message body

[DeliZhang]

BPO	25553
Nosy	@warsaw, @bitdancer
Files	test.eml: sample mail causing problem smtplib.py: smtplib.py changes base on python 2.7.10

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-10.00:51:28.103>
created_at = <Date 2015-11-05.06:51:55.043>
labels = ['type-bug', 'library', 'expert-email']
title = 'smtpd strips final carraige return from received message body'
updated_at = <Date 2018-01-10.00:51:28.102>
user = 'https://bugs.python.org/DeliZhang'

bugs.python.org fields:

activity = <Date 2018-01-10.00:51:28.102>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-10.00:51:28.103>
closer = 'barry'
components = ['Library (Lib)', 'email']
creation = <Date 2015-11-05.06:51:55.043>
creator = 'Deli Zhang'
dependencies = []
files = ['40944', '40945']
hgrepos = []
issue_num = 25553
keywords = []
message_count = 6.0
messages = ['254086', '254087', '254127', '254185', '254194', '309747']
nosy_count = 3.0
nosy_names = ['barry', 'r.david.murray', 'Deli Zhang']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25553'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[DeliZhang]

It's well known that in quoted-printable encoding the "CRLF" can be encoded to "=CRLF".
For example, in attachment file test.eml, the last line is "test message.=CRLF"
But after the mail is sent via SMTP and received by mail server (e.g. postfix), the last line will become "test message.=",
then after decoding, you will see the unnecessary char "=" shown following "test message.".

The problem is caused by below code:
------------------------------------------------

class SMTP:
    ...
    def data(self, msg):
        ...
            q = quotedata(msg)
            if q[-2:] != CRLF:
                q = q + CRLF
            q = q + "." + CRLF
            self.send(q)
        ...

---

Before it sends the message q, it will try to append the end-of-data sequence "<CRLF>.<CRLF>".
As the implement, it checks whether there is "<CRLF>" in the message end, if yes then just need to append ".<CRLF>".
It looks rigorous and efficient, but it does not consider how mail server handle it.
As we know mail server will remove end-of-data sequence directly, and the left message is treat as mail data.

Thus the corresponding action should be taken on SMTP client side,
it's to say no need to check and just append end-of-data sequence here:
------------------------------------------------

class SMTP:
    ...
    def data(self, msg):
        ...
            q = quotedata(msg)
            q = q + CRLF + "." + CRLF
            self.send(q)
        ...

---

[DeliZhang]

Correct the action of appending the end-of-data sequence "<CRLF>.<CRLF>"

[bitdancer]

RFC 2812 says:

Note that the first <CRLF> of this terminating sequence is also the <CRLF> that ends the final line of the data (message text)

So, smtplib is correct. If you have a server that is not respecting this, then that server is out of compliance and there isn't anything we can do about it.

However, I don't think that is your problem. = at the end of a line actually represents a "soft carriage return", which means one that is *eliminated* in the decoded output. If you will read section 6.7 of rfc 2045, specifically notes (2) and (3) in the second block of numbered paragraphs, you will see that an 'ultimate' = (an = at the end of an encoded block, with or without a CRLF after it), such as you have in your sample, is illegal. Further, the recommended recovery action if one is seen while decoding is to leave the = in the decoded output, just as you are observing happening.

So, there is no bug here except in your message :)

[DeliZhang]

I can understand you, while could you please consider below fact:
Once our SMTP server module smtpd.py receives the sample mail, it will remove the end-of-data sequence, that makes the "=" become the last char of mail data. I think it's inconsistent to our SMTP client module smtplib.py.

If we can just add "<CRLF>.<CRLF>" following mail data like postfix, which is influential amd authoritative in SMTP field, that will make things simple and will not make any trouble in reality situation.

I just advise this, if you think no need then I can compromise.
Thanks.

[bitdancer]

That does indeed appear to be a bug in smtpd.py.

Even if postfix's client mode does do an unconditional add of an extra newline, it is wrong according to the RFC, so I don't think that we'd change smtplib. Especially since we've had no other reports of the current code causing problems (the only problem case you have reported is against smtpd.py, which is a bug in smtpd.py).

[warsaw]

I'm closing this as won't fix since smtpd.py is deprecated and likely won't see much future improvements. Please take a look at aiosmtpd as a much better third party replacement.