Deprecate asyncore/asynchat

[gvanrossum]

BPO	25002
Nosy	@gvanrossum, @warsaw, @brettcannon, @rhettinger, @vstinner, @ericvsmith, @giampaolo, @bitdancer, @vadmium, @serhiy-storchaka, @dstufft, @Mariatta
Dependencies	bpo-25008: Deprecate smtpd (based on deprecated asyncore/asynchat)
Files	issue25002.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-04-19.21:55:22.038>
created_at = <Date 2015-09-04.20:48:03.817>
labels = ['3.7', 'type-feature', 'library']
title = 'Deprecate asyncore/asynchat'
updated_at = <Date 2017-04-20.01:12:35.577>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2017-04-20.01:12:35.577>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2017-04-19.21:55:22.038>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2015-09-04.20:48:03.817>
creator = 'gvanrossum'
dependencies = ['25008']
files = ['45210']
hgrepos = []
issue_num = 25002
keywords = ['patch']
message_count = 37.0
messages = ['249829', '249830', '249831', '249833', '249855', '249873', '249882', '249912', '250109', '250150', '250151', '250214', '250223', '250232', '250233', '250234', '250239', '250260', '279352', '279416', '279417', '279424', '279429', '279436', '279437', '279438', '279440', '279442', '279444', '279450', '279464', '279469', '279477', '279478', '291909', '291912', '291929']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'barry', 'brett.cannon', 'rhettinger', 'vstinner', 'eric.smith', 'giampaolo.rodola', 'r.david.murray', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'dstufft', 'Mariatta']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25002'
versions = ['Python 3.6', 'Python 3.7']

[gvanrossum]

Now that we've got asyncio in two releases (3.4 and 3.5) we should start deprecating asyncore and asynchat. There isn't much use of these in the stdlib (smtpd.py uses them, and a bunch of tests) and we should probably rewrite those to use asyncio. Maybe smtpd.py can be deprecated itself.

[dstufft]

I'm all for deprecating asyncore/asynchat but should deprecating them wait until asyncio is no longer provisional?

[gvanrossum]

Yes, in 3.6 asyncio will no longer be provisional and we can start deprecating async{ore,hat}. Which is why I marked this bug with 3.6.

[vstinner]

The documentation of both modules begin with the following note:

"This module exists for backwards compatibility only. For new code we recommend using asyncio."

What do you mean by deprecating the module? Emit a PendingDeprecationWarning in Python 3.6 and emit a DeprecationWarning in Python 3.7?

Maybe smtpd.py can be deprecated itself.

Maybe it can be fun to rewrite the module using asyncio, but I'm not convinced that a SMTP server in the Python stdlib is super useful.

(A HTTP server *is* very useful, it saved my life multiple times when I worked on embedded devices without any server available to transfer files.)

[gvanrossum]

Ideally these modules should emit a deprecation warning starting in 3.6 (when asyncio is no longer provisional) and we should strive to delete them per 3.8.

If nobody rewrites smptd.py using asyncio it should be deleted at the same time.

[ericvsmith]

I use smtpd.py for testing my code. But it's not such a big deal that I couldn't live without it. If I have some time to burn, I might convert it to asyncio so I can continue to use it.

[bitdancer]

smtpd is used for testing smtplib. It is also used in test_logging. I would object to having it removed, but I suspect we can manage to rewrite it by 3.8. Maybe Eric and I can collaborate on it.

[brettcannon]

I opened http://bugs.python.org/issue25008 to discuss deprecating smtpd since this issue is about asyncore/asynchat.

[warsaw]

On Sep 04, 2015, at 08:55 PM, STINNER Victor wrote:

Maybe it can be fun to rewrite the module using asyncio, but I'm not
convinced that a SMTP server in the Python stdlib is super useful.

As I mentioned in bpo-25008, removing smtpd would be a hardship for myself,
and I don't think I'm alone. I would however love to have (and maybe even
work on <wink>) an asyncio-based version. I do think it is useful to have an
SMTP server in the stdlib - and actually, I'd also like to have an LMTP server
there too. But I think we can take baby steps toward this by some folks
getting together and writing a third party asyncio-based module, releasing
this on PyPI and eventually leading to its inclusion in Python down the road.
I think there's no hurry to remove smtpd, but it can be deprecated-in-place
like optparse, as someone else mentioned.

[rhettinger]

Now that we've got asyncio in two releases (3.4 and 3.5)
we should start deprecating asyncore and asynchat.

If we care about making it easier for organizations to move from Python 2 to Python 3, we should develop a strong aversion to deprecating modules that have been around for a long time and aren't broken.

In addition, we should give weight and support to the numerous projects that are trying to have a code base the runs in both Python 2 and Python 3 (hence the astonishing success of the six module).

Already my tooling for testing Python 3.6 is broken because some core dev aggressively removed functions from the inspect module that were deprecated in favor of using Signature objects. That broke the popular Hypothesis testing tool which was carefully written to work for both Python 2 and Python 3. In order to work again that tool will have to be modified to copy in the old code that was just taken out.

don't-leave-your-users-behind-ly yours ...

[gvanrossum]

Then can we at least close any feature requests for asyncore/asynchat as wontfix? (And porting smtpd.py to asyncio is still a good idea.)

[bitdancer]

Yes, stopping changing asyncore/asynchat also means it becomes a stable target for people who *are* using it for 2/3 code. We may have effectively done this already (without closing the open issues): the last asyncore-specific change (as opposed to library-wide changes that also hit asyncore) was a ResourceWarning added by Victor in June of 2014. Likewise for asynchat.

[vstinner]

If you care of smooth python 2 => python 3 migration, I suggest to
deprecate smtpd and remove it later, when asyncore & asynchat will
also be removed (ex: in Python 4?).

I suggest to write a *new* SMTP server module using asyncio. As I
explained, the API will be very different for the user of the API.

[brettcannon]

Victor's suggestion also aligns with my thinking on the subject as well: deprecate the modules but simply leave them sitting there for compatibility reasons until Python 4 of whenever we feel like the Python 2 transition is done for those that will ever bother making the switch. That way people's expectations are in the proper alignment of where things will end up and code won't break if it straddles Python 2 & 3 short of having to silence a deprecation warning. I think this is a good use of PendingDeprecationWarning and the message both in the deprecation and the docs can make it clear that the removal date is not planned but you should avoid using the module in new code.

[vstinner]

Ideally these modules should emit a deprecation warning starting in 3.6

Is it ok to add the PendingDeprecationWarning in Python 3.5.1?

[gvanrossum]

Is it ok to add the PendingDeprecationWarning in Python 3.5.1?

I prefer to wait until 3.6. A bugfix release should not rock the boat, it should not make your code emit new warnings.

[rhettinger]

In addition to a PendingDeprecationWarning in 3.6, it might be nice to put a note in the asyncore/asynchat docs like we did for optparse.

[BreamoreBoy]

IIRC I flagged up oustanding asyncore/asynchat issues months or even years back and Victor closed a lot down, so there shouldn't be many left to deal with.

[Mariatta]

Hi,

Attached is the patch that adds pending deprecation warnings to asyncore and asynchat. Please review. Thanks :)

[gvanrossum]

Applied:

remote: notified python-checkins@python.org of incoming changeset bb23770f82f1
remote: notified python-checkins@python.org of incoming changeset 3b8dfe6f5bcb

[python-dev]

New changeset bb23770f82f1 by Guido van Rossum in branch '3.6':
bpo-25002: Deprecate asyncore/asynchat. Patch by Mariatta.
https://hg.python.org/cpython/rev/bb23770f82f1

New changeset 3b8dfe6f5bcb by Guido van Rossum in branch 'default':
bpo-25002: Deprecate asyncore/asynchat. Patch by Mariatta. (3.6->3.7)
https://hg.python.org/cpython/rev/3b8dfe6f5bcb

[serhiy-storchaka]

This change broke buildbots.

$ ./python -We -m test.regrtest test_os 
Run tests sequentially
0:00:00 [1/1] test_os
test test_os crashed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 151, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/home/serhiy/py/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 677, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/home/serhiy/py/cpython/Lib/test/test_os.py", line 5, in <module>
    import asynchat
  File "/home/serhiy/py/cpython/Lib/asynchat.py", line 48, in <module>
    import asyncore
  File "/home/serhiy/py/cpython/Lib/asyncore.py", line 65, in <module>
    PendingDeprecationWarning, stacklevel=2)
PendingDeprecationWarning: asyncore module is deprecated in 3.6. Use asyncio instead.

test_os failed

1 test failed:
test_os

Total duration: 59 ms
Tests result: FAILURE

[gvanrossum]

Sorry about that. Should I roll it back or is there a way to make the test pass (expect this deprecation)?

[serhiy-storchaka]

I think replacing

    import asynchat
    import asyncore

with

    with warnings.catch_warnings():
        warnings.simplefilter('ignore', PendingDeprecationWarning)
        import asynchat
        import asyncore

can help.

asynchat and asyncore are used in several tests.

In long term asynchat and asyncore should be replaced with alternatives.

[Mariatta]

Sorry about this Serhiy. I can work on another patch based on your code snippet later today.

Should I create a different ticket about replacing asyncore and asynchat?

[serhiy-storchaka]

Should I create a different ticket about replacing asyncore and asynchat?

This may require several tickets, one per case.

[serhiy-storchaka]

Ah, asyncore/asynchat are used in smtpd!

It looks to me that we should first write asyncio alternative to smtpd, then deprecate smtpd, and only after this we can deprecate asyncore/asynchat.

[Mariatta]

Thanks Serhiy,

I created these two tickets:
https://bugs.python.org/issue28534
https://bugs.python.org/issue28533

[bitdancer]

The alternative has already been written: aiosmtpd, in the aiolibs project. The question is should it be added to the stdlib...

[gvanrossum]

Serhiy, so should I revert this patch for now? Or are the silent
deprecation warnings outside the test suite okay? (In that case, maybe
Mariatta can upload a patch?)

[vstinner]

R. David Murray added the comment:

The alternative has already been written: aiosmtpd, in the aiolibs
project. The question is should it be added to the stdlib...

I suggest to keep it on PyPI to keep fast releases.

[vadmium]

Perhaps it is okay to keep the documentation changes, but I think either the library changes should be reverted or worked around where the modules are still in use.

I normally run the tests with -Werror, and the failures I get are:

• test_ssl, test_smtplib, test_poplib, test_logging, test_ftplib, test_support: tests use asyncore
• test_os: test uses asynchat (When you import asynchat, the first error complains about importing asyncore, there are actually two warnings)
• test_all: This seems to ignore DeprecationWarning; perhaps an exception for PendingDeprecationWarning should also be added?
• test_asyncore and test_asynchat: Obviously these have to still test the modules, so they should anticipate the warnings, perhaps using Serhiy’s code
• test_smtpd: smtpd module itself uses asyncore; see bpo-25008

[python-dev]

New changeset 6eb3312a9a16 by Guido van Rossum in branch '3.6':
Issue bpo-25002: Back out asyncore/asynchat deprecation.
https://hg.python.org/cpython/rev/6eb3312a9a16

New changeset 2879185bc511 by Guido van Rossum in branch 'default':
Issue bpo-25002: Back out asyncore/asynchat deprecation. (3.6->3.7)
https://hg.python.org/cpython/rev/2879185bc511

[gvanrossum]

OK, backed out the code changes, kept the docs.

[vstinner]

I suggest to close this issue and instead work on the the issue bpo-28533: modify code using asyncore to use something else.

[gvanrossum]

Sounds good.

[warsaw]

For the archaeologists of the future, smtpd.py is also deprecated in favor of aiosmtpd.

http://aiosmtpd.readthedocs.io/en/latest/