Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate asyncore/asynchat #69190

Closed
gvanrossum opened this issue Sep 4, 2015 · 37 comments
Closed

Deprecate asyncore/asynchat #69190

gvanrossum opened this issue Sep 4, 2015 · 37 comments
Labels
3.7 stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@gvanrossum
Copy link
Member

gvanrossum commented Sep 4, 2015

BPO 25002
Nosy @gvanrossum, @warsaw, @brettcannon, @rhettinger, @vstinner, @ericvsmith, @giampaolo, @bitdancer, @vadmium, @serhiy-storchaka, @dstufft, @Mariatta
Dependencies
  • bpo-25008: Deprecate smtpd (based on deprecated asyncore/asynchat)
  • Files
  • issue25002.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-04-19.21:55:22.038>
    created_at = <Date 2015-09-04.20:48:03.817>
    labels = ['3.7', 'type-feature', 'library']
    title = 'Deprecate asyncore/asynchat'
    updated_at = <Date 2017-04-20.01:12:35.577>
    user = 'https://github.com/gvanrossum'

    bugs.python.org fields:

    activity = <Date 2017-04-20.01:12:35.577>
    actor = 'barry'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-04-19.21:55:22.038>
    closer = 'gvanrossum'
    components = ['Library (Lib)']
    creation = <Date 2015-09-04.20:48:03.817>
    creator = 'gvanrossum'
    dependencies = ['25008']
    files = ['45210']
    hgrepos = []
    issue_num = 25002
    keywords = ['patch']
    message_count = 37.0
    messages = ['249829', '249830', '249831', '249833', '249855', '249873', '249882', '249912', '250109', '250150', '250151', '250214', '250223', '250232', '250233', '250234', '250239', '250260', '279352', '279416', '279417', '279424', '279429', '279436', '279437', '279438', '279440', '279442', '279444', '279450', '279464', '279469', '279477', '279478', '291909', '291912', '291929']
    nosy_count = 13.0
    nosy_names = ['gvanrossum', 'barry', 'brett.cannon', 'rhettinger', 'vstinner', 'eric.smith', 'giampaolo.rodola', 'r.david.murray', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'dstufft', 'Mariatta']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue25002'
    versions = ['Python 3.6', 'Python 3.7']

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Sep 4, 2015

    Now that we've got asyncio in two releases (3.4 and 3.5) we should start deprecating asyncore and asynchat. There isn't much use of these in the stdlib (smtpd.py uses them, and a bunch of tests) and we should probably rewrite those to use asyncio. Maybe smtpd.py can be deprecated itself.

    @dstufft
    Copy link
    Member

    dstufft commented Sep 4, 2015

    I'm all for deprecating asyncore/asynchat but should deprecating them wait until asyncio is no longer provisional?

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Sep 4, 2015

    Yes, in 3.6 asyncio will no longer be provisional and we can start deprecating async{ore,hat}. Which is why I marked this bug with 3.6.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 4, 2015

    The documentation of both modules begin with the following note:

    "This module exists for backwards compatibility only. For new code we recommend using asyncio."

    What do you mean by deprecating the module? Emit a PendingDeprecationWarning in Python 3.6 and emit a DeprecationWarning in Python 3.7?

    Maybe smtpd.py can be deprecated itself.

    Maybe it can be fun to rewrite the module using asyncio, but I'm not convinced that a SMTP server in the Python stdlib is super useful.

    (A HTTP server *is* very useful, it saved my life multiple times when I worked on embedded devices without any server available to transfer files.)

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Sep 4, 2015

    Ideally these modules should emit a deprecation warning starting in 3.6 (when asyncio is no longer provisional) and we should strive to delete them per 3.8.

    If nobody rewrites smptd.py using asyncio it should be deleted at the same time.

    @ericvsmith
    Copy link
    Member

    ericvsmith commented Sep 5, 2015

    I use smtpd.py for testing my code. But it's not such a big deal that I couldn't live without it. If I have some time to burn, I might convert it to asyncio so I can continue to use it.

    @bitdancer
    Copy link
    Member

    bitdancer commented Sep 5, 2015

    smtpd is used for testing smtplib. It is also used in test_logging. I would object to having it removed, but I suspect we can manage to rewrite it by 3.8. Maybe Eric and I can collaborate on it.

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 5, 2015

    I opened http://bugs.python.org/issue25008 to discuss deprecating smtpd since this issue is about asyncore/asynchat.

    @warsaw
    Copy link
    Member

    warsaw commented Sep 7, 2015

    On Sep 04, 2015, at 08:55 PM, STINNER Victor wrote:

    Maybe it can be fun to rewrite the module using asyncio, but I'm not
    convinced that a SMTP server in the Python stdlib is super useful.

    As I mentioned in bpo-25008, removing smtpd would be a hardship for myself,
    and I don't think I'm alone. I would however love to have (and maybe even
    work on <wink>) an asyncio-based version. I do think it is useful to have an
    SMTP server in the stdlib - and actually, I'd also like to have an LMTP server
    there too. But I think we can take baby steps toward this by some folks
    getting together and writing a third party asyncio-based module, releasing
    this on PyPI and eventually leading to its inclusion in Python down the road.
    I think there's no hurry to remove smtpd, but it can be deprecated-in-place
    like optparse, as someone else mentioned.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Sep 8, 2015

    Now that we've got asyncio in two releases (3.4 and 3.5)
    we should start deprecating asyncore and asynchat.

    If we care about making it easier for organizations to move from Python 2 to Python 3, we should develop a strong aversion to deprecating modules that have been around for a long time and aren't broken.

    In addition, we should give weight and support to the numerous projects that are trying to have a code base the runs in both Python 2 and Python 3 (hence the astonishing success of the six module).

    Already my tooling for testing Python 3.6 is broken because some core dev aggressively removed functions from the inspect module that were deprecated in favor of using Signature objects. That broke the popular Hypothesis testing tool which was carefully written to work for both Python 2 and Python 3. In order to work again that tool will have to be modified to copy in the old code that was just taken out.

    don't-leave-your-users-behind-ly yours ...

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Sep 8, 2015

    Then can we at least close any feature requests for asyncore/asynchat as wontfix? (And porting smtpd.py to asyncio is still a good idea.)

    @bitdancer
    Copy link
    Member

    bitdancer commented Sep 8, 2015

    Yes, stopping changing asyncore/asynchat also means it becomes a stable target for people who *are* using it for 2/3 code. We may have effectively done this already (without closing the open issues): the last asyncore-specific change (as opposed to library-wide changes that also hit asyncore) was a ResourceWarning added by Victor in June of 2014. Likewise for asynchat.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    If you care of smooth python 2 => python 3 migration, I suggest to
    deprecate smtpd and remove it later, when asyncore & asynchat will
    also be removed (ex: in Python 4?).

    I suggest to write a *new* SMTP server module using asyncio. As I
    explained, the API will be very different for the user of the API.

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 8, 2015

    Victor's suggestion also aligns with my thinking on the subject as well: deprecate the modules but simply leave them sitting there for compatibility reasons until Python 4 of whenever we feel like the Python 2 transition is done for those that will ever bother making the switch. That way people's expectations are in the proper alignment of where things will end up and code won't break if it straddles Python 2 & 3 short of having to silence a deprecation warning. I think this is a good use of PendingDeprecationWarning and the message both in the deprecation and the docs can make it clear that the removal date is not planned but you should avoid using the module in new code.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    Ideally these modules should emit a deprecation warning starting in 3.6

    Is it ok to add the PendingDeprecationWarning in Python 3.5.1?

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Sep 8, 2015

    Is it ok to add the PendingDeprecationWarning in Python 3.5.1?

    I prefer to wait until 3.6. A bugfix release should not rock the boat, it should not make your code emit new warnings.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Sep 8, 2015

    In addition to a PendingDeprecationWarning in 3.6, it might be nice to put a note in the asyncore/asynchat docs like we did for optparse.

    @BreamoreBoy
    Copy link
    Mannequin

    BreamoreBoy mannequin commented Sep 8, 2015

    IIRC I flagged up oustanding asyncore/asynchat issues months or even years back and Victor closed a lot down, so there shouldn't be many left to deal with.

    @Mariatta
    Copy link
    Sponsor Member

    Mariatta commented Oct 25, 2016

    Hi,

    Attached is the patch that adds pending deprecation warnings to asyncore and asynchat. Please review. Thanks :)

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Oct 25, 2016

    Applied:

    remote: notified python-checkins@python.org of incoming changeset bb23770f82f1
    remote: notified python-checkins@python.org of incoming changeset 3b8dfe6f5bcb

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 25, 2016

    New changeset bb23770f82f1 by Guido van Rossum in branch '3.6':
    bpo-25002: Deprecate asyncore/asynchat. Patch by Mariatta.
    https://hg.python.org/cpython/rev/bb23770f82f1

    New changeset 3b8dfe6f5bcb by Guido van Rossum in branch 'default':
    bpo-25002: Deprecate asyncore/asynchat. Patch by Mariatta. (3.6->3.7)
    https://hg.python.org/cpython/rev/3b8dfe6f5bcb

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Oct 25, 2016

    This change broke buildbots.

    $ ./python -We -m test.regrtest test_os 
    Run tests sequentially
    0:00:00 [1/1] test_os
    test test_os crashed -- Traceback (most recent call last):
      File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 151, in runtest_inner
        the_module = importlib.import_module(abstest)
      File "/home/serhiy/py/cpython/Lib/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 978, in _gcd_import
      File "<frozen importlib._bootstrap>", line 961, in _find_and_load
      File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 677, in exec_module
      File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
      File "/home/serhiy/py/cpython/Lib/test/test_os.py", line 5, in <module>
        import asynchat
      File "/home/serhiy/py/cpython/Lib/asynchat.py", line 48, in <module>
        import asyncore
      File "/home/serhiy/py/cpython/Lib/asyncore.py", line 65, in <module>
        PendingDeprecationWarning, stacklevel=2)
    PendingDeprecationWarning: asyncore module is deprecated in 3.6. Use asyncio instead.

    test_os failed

    1 test failed:
    test_os

    Total duration: 59 ms
    Tests result: FAILURE

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Oct 25, 2016

    Sorry about that. Should I roll it back or is there a way to make the test pass (expect this deprecation)?

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Oct 25, 2016

    I think replacing

        import asynchat
        import asyncore

    with

        with warnings.catch_warnings():
            warnings.simplefilter('ignore', PendingDeprecationWarning)
            import asynchat
            import asyncore

    can help.

    asynchat and asyncore are used in several tests.

    In long term asynchat and asyncore should be replaced with alternatives.

    @Mariatta
    Copy link
    Sponsor Member

    Mariatta commented Oct 25, 2016

    Sorry about this Serhiy. I can work on another patch based on your code snippet later today.

    Should I create a different ticket about replacing asyncore and asynchat?

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Oct 25, 2016

    Should I create a different ticket about replacing asyncore and asynchat?

    This may require several tickets, one per case.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir 3.7 labels Oct 25, 2016
    @serhiy-storchaka serhiy-storchaka added the type-feature A feature request or enhancement label Oct 25, 2016
    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Oct 25, 2016

    Ah, asyncore/asynchat are used in smtpd!

    It looks to me that we should first write asyncio alternative to smtpd, then deprecate smtpd, and only after this we can deprecate asyncore/asynchat.

    @Mariatta
    Copy link
    Sponsor Member

    Mariatta commented Oct 25, 2016

    Thanks Serhiy,

    I created these two tickets:
    https://bugs.python.org/issue28534
    https://bugs.python.org/issue28533

    @bitdancer
    Copy link
    Member

    bitdancer commented Oct 25, 2016

    The alternative has already been written: aiosmtpd, in the aiolibs project. The question is should it be added to the stdlib...

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Oct 25, 2016

    Serhiy, so should I revert this patch for now? Or are the silent
    deprecation warnings outside the test suite okay? (In that case, maybe
    Mariatta can upload a patch?)

    @vstinner
    Copy link
    Member

    vstinner commented Oct 25, 2016

    R. David Murray added the comment:

    The alternative has already been written: aiosmtpd, in the aiolibs
    project. The question is should it be added to the stdlib...

    I suggest to keep it on PyPI to keep fast releases.

    @vadmium
    Copy link
    Member

    vadmium commented Oct 26, 2016

    Perhaps it is okay to keep the documentation changes, but I think either the library changes should be reverted or worked around where the modules are still in use.

    I normally run the tests with -Werror, and the failures I get are:

    • test_ssl, test_smtplib, test_poplib, test_logging, test_ftplib, test_support: tests use asyncore
    • test_os: test uses asynchat (When you import asynchat, the first error complains about importing asyncore, there are actually two warnings)
    • test_all: This seems to ignore DeprecationWarning; perhaps an exception for PendingDeprecationWarning should also be added?
    • test_asyncore and test_asynchat: Obviously these have to still test the modules, so they should anticipate the warnings, perhaps using Serhiy’s code
    • test_smtpd: smtpd module itself uses asyncore; see bpo-25008

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 26, 2016

    New changeset 6eb3312a9a16 by Guido van Rossum in branch '3.6':
    Issue bpo-25002: Back out asyncore/asynchat deprecation.
    https://hg.python.org/cpython/rev/6eb3312a9a16

    New changeset 2879185bc511 by Guido van Rossum in branch 'default':
    Issue bpo-25002: Back out asyncore/asynchat deprecation. (3.6->3.7)
    https://hg.python.org/cpython/rev/2879185bc511

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Oct 26, 2016

    OK, backed out the code changes, kept the docs.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 19, 2017

    I suggest to close this issue and instead work on the the issue bpo-28533: modify code using asyncore to use something else.

    @gvanrossum
    Copy link
    Member Author

    gvanrossum commented Apr 19, 2017

    Sounds good.

    @warsaw
    Copy link
    Member

    warsaw commented Apr 20, 2017

    For the archaeologists of the future, smtpd.py is also deprecated in favor of aiosmtpd.

    http://aiosmtpd.readthedocs.io/en/latest/

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests