Deprecate smtpd (based on deprecated asyncore/asynchat)

[brettcannon]

BPO	25008
Nosy	@warsaw, @brettcannon, @rhettinger, @terryjreedy, @vstinner, @ericvsmith, @giampaolo, @ned-deily, @bitdancer
PRs	bpo-25008: Deprecate smtpd and point to aiosmtpd #274 bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) #278 Cherry pick bpo-25008 to 3.5 #279 Cherry pick bpo-25008 to 3.4 #280 [Do Not Merge] Sample of CPython life with blurb. #703 [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	aiosmtpd.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/warsaw'
closed_at = <Date 2017-02-24.20:32:41.162>
created_at = <Date 2015-09-05.17:08:13.926>
labels = ['3.7', 'docs']
title = 'Deprecate smtpd (based on deprecated asyncore/asynchat)'
updated_at = <Date 2019-05-10.18:11:02.809>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2019-05-10.18:11:02.809>
actor = 'ned.deily'
assignee = 'barry'
closed = True
closed_date = <Date 2017-02-24.20:32:41.162>
closer = 'barry'
components = ['Documentation']
creation = <Date 2015-09-05.17:08:13.926>
creator = 'brett.cannon'
dependencies = []
files = ['40605']
hgrepos = []
issue_num = 25008
keywords = ['patch']
message_count = 18.0
messages = ['249911', '249913', '249935', '249942', '249953', '250019', '250110', '250111', '250149', '250510', '251736', '252285', '253097', '288391', '290408', '290409', '290410', '290411']
nosy_count = 10.0
nosy_names = ['barry', 'brett.cannon', 'rhettinger', 'terry.reedy', 'vstinner', 'eric.smith', 'giampaolo.rodola', 'ned.deily', 'r.david.murray', 'neic']
pr_nums = ['274', '278', '279', '280', '703', '552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue25008'
versions = ['Python 3.7']

[brettcannon]

As mentioned in passing in issue bpo-25002, perhaps smtpd should be deprecated or at least be updated to use asyncio.

[brettcannon]

I should mention one of the motivations behind deprecating smtpd is that no one should be using that module to run an SMTP server and it doesn't provide utility like http.server does and so we probably should drop the code to prevent anyone accidentally using it and not have to maintain it.

[rhettinger]

Bringing in Eric V Smith's comment in
http://bugs.python.org/issue25002#msg249873 :

'''
I use smtpd.py for testing my code. But it's not such a big deal that I couldn't live without it. If I have some time to burn, I might convert it to asyncio so I can continue to use it.

'''

Bringing in R David Murray's comment in http://bugs.python.org/issue25002#msg249882 :

'''smtpd is used for testing smtplib. It is also used in test_logging. I would object to having it removed, but I suspect we can manage to rewrite it by 3.8. Maybe Eric and I can collaborate on it.
'''

[rhettinger]

FWIW, I would like to see the module live on (after conversion to asyncio). Doug Hellman's example at https://pymotw.com/2/smtpd/ makes for a nice demonstration of Python's capabilities.

[warsaw]

Removing smtpd would definitely be a hardship for me right now, probably for obvious reasons. I use it in testing frameworks, and even wrote a library called lazr.smtptest that is built around smtpd. In Mailman, we have an LMTP server built on smtpd that implements *the* official way of getting mail into the system from an upstream mail server. Mailman 3 core FWIW is Python 3.4 and 3.5 compatible.

That all being said, I've also been thinking lately about an asyncio-based reimplementation of SMTP and LMTP. asyncore/chat and smtpd are not the easiest modules to work with, extend, understand, or debug. I'd be very happy to collaborate with folks on an asyncio-based version. Maybe we can put that out as a third party module, with a long term plan on pulling it into the stdlib and promoting it instead of smtpd.

[bitdancer]

I believe that smtpd is used for a *lot* of testing code (anyone who is writing smtp clients in python, pretty much, and probably some writing in other languages). I really don't see the rationale for removing it, assuming we can rewrite it "in time".

On the other hand, are we really going to *remove* asyncore/asynchat? Or is this a "deprecation in place" like optparse? If the latter then I don't really care if the same is done to smtpd :) If the former, why, if we've kept optparse?

[vstinner]

Even if it may be possible to support asyncore/asynchat and asyncio at the same time, I would prefer to drop asyncore/asynchat support to simplify the code.

The problem is that the API will be very different.

asyncio has a different design. You don't instanciate a class to open a network connection, but you have to call "yield from loop.create_connection()" in a coroutine.

We might provide helper blocking functions starting the server for us and then running the event loop until CTRL+c is pressed (or another event to stop to stop the loop).

Rewriting smtpd with asyncio would allow to use it in an asyncio application. In 2015, I expect to see more and more applications using asyncio than asyncore/asynchat.

Can we modify the issue title to "Rewrite smtpd with asyncio"?

[warsaw]

On Sep 07, 2015, at 02:26 PM, STINNER Victor wrote:

Can we modify the issue title to "Rewrite smtpd with asyncio"?

Sure, although I'm currently thinking it's best to go third party until the
API and implementation settle down.

[rhettinger]

Unless there is a drop-in replacement, I don't think this should be deprecated or removed. Our biggest problem in the Python world is getting organizations using Python 2 to switch to Python 3. Deprecating modules that have been a part of the firmament for a long time works against that goal.

Side note: a quick search shows that people are still writing SMTPD modules in other languages (i.e. Go now has one)

[terryjreedy]

http://nullege.com/codes/search?cq=import+smtpd
shows 135 hits. FWIW, import+asyncore get 972.

[bitdancer]

Here is a proof of concept port of smtpd to asyncio. Obviously I'm not suggesting we commit this patch (it isn't complete...the tests don't run). Instead I'm posting it as an outgrowth the of the sprint that was conducted today. The DC team took a different tack, so I'm not submitting this as a patch to their repo, and since it is useful to look at the diff against default, it seems appropriate to post it here.

The advantage of this port is that it uses the existing smtpd protocol code, which is the "field tested" code. Of course, I did have to change how the data is turned from bytes into lines, so I may have introduced some bugs.

As discussed, *using* an asyncio based smtpd has a different calling API than using the asynchat one. However, the *pattern* is similar, so it may not be crazy to replace smtpd with the cleaned up version of this patch, exactly because the number of places this is used is relatively small and mostly in test code. IMO it's a better option than deleting smtpd from the stdlib.

The code as is can be run using its CLI. I tested it via telnet using the DebuggingServer and hand coding an email...for some reason I can't currently import smtplib (Enum errors in socket), so I didn't test it with smtplib, but it should work.

[warsaw]

I'm liking RDM's port of smtpd.py. I've done some clean ups, bug fixes, and added some debugging. See this branch over on gitlab:

https://gitlab.com/python-smtpd-hackers/aiosmtpd/tree/issue25508

Try python3 server.py then run client.py in another terminal (it uses smtplib) or just telnet.

I plan to continue pursuing this approach; please comment over there.

[warsaw]

This has come along pretty well. The project is managed here:

https://gitlab.com/python-smtpd-hackers/aiosmtpd

and documentation here:

http://aiosmtpd.readthedocs.org/en/latest/

[warsaw]

aiosmtpd is coming along nicely:

http://aiosmtpd.readthedocs.io/en/latest/

We'll soon have a 1.0 release. Still, I don't think it's worth pulling this into the stdlib. But we could silently deprecate it and point to aiosmtpd in the docs.

[warsaw]

New changeset 66b5092 by Barry Warsaw in branch '3.5':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#279)
66b5092

[warsaw]

New changeset 0a1b656 by Barry Warsaw in branch '3.6':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#278)
0a1b656

[warsaw]

New changeset f37b0cb by Barry Warsaw in branch '3.4':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274) (#280)
f37b0cb

[vstinner]

New changeset fba79dc by Victor Stinner (Barry Warsaw) in branch 'master':
bpo-25008: Deprecate smtpd and point to aiosmtpd (#274)
fba79dc