Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Server.serve_forever and corresponding APIs #76843

Closed
1st1 opened this issue Jan 25, 2018 · 7 comments
Closed

Implement Server.serve_forever and corresponding APIs #76843

1st1 opened this issue Jan 25, 2018 · 7 comments
Labels
3.7 expert-asyncio type-feature

Comments

@1st1
Copy link
Member

@1st1 1st1 commented Jan 25, 2018

BPO 32662
Nosy @vstinner, @asvetlov, @1st1
PRs
  • #5312
  • #5338
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-05-29.17:03:12.778>
    created_at = <Date 2018-01-25.06:44:24.572>
    labels = ['type-feature', '3.7', 'expert-asyncio']
    title = 'Implement Server.serve_forever and corresponding APIs'
    updated_at = <Date 2018-05-29.17:03:12.777>
    user = 'https://github.com/1st1'

    bugs.python.org fields:

    activity = <Date 2018-05-29.17:03:12.777>
    actor = 'yselivanov'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-05-29.17:03:12.778>
    closer = 'yselivanov'
    components = ['asyncio']
    creation = <Date 2018-01-25.06:44:24.572>
    creator = 'yselivanov'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 32662
    keywords = ['patch']
    message_count = 7.0
    messages = ['310657', '310717', '310740', '310748', '310749', '310750', '310786']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'asvetlov', 'yselivanov']
    pr_nums = ['5312', '5338']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue32662'
    versions = ['Python 3.7']

    @1st1
    Copy link
    Member Author

    @1st1 1st1 commented Jan 25, 2018

    As discussed, we want to make Server objects more usable in async/await code and more compatible with asyncio.run.

    This is also needed to handle a use case when two or more servers are created and need to start listening at the same time.

    We propose to:

    1. Add a new bool flag defaulting to True to loop.create_server and loop.create_unix_server: start_serving. By default, loop will return a server that is already accepting connections. When start_serving is set to False, create_server and create_unix_server will return a server that will not listen on its sockets.

    2. A new idempotent Server.start_serving() method can be used to make server listen on its sockets (useful when a server object was created with start_serving=False).

    3. A new Server.serve_forever() method that calls start_serving() and blocks forever, until cancelled. When cancelled, it closes its server object.

    4. A new Server.is_serving() method. This is useful to introspect a server object in unittests.

    5. Server objects should be async context managers. Server.__aexit__ should close the server and await on Server.wait_closed().

    With these new APIs, the following pattern becomes possible:

        async def main():
            srv = await asyncio.start_server(...)
            async with srv:
                await srv.serve_forever()
    
        asyncio.run(main())

    @1st1 1st1 added 3.7 expert-asyncio type-feature labels Jan 25, 2018
    @1st1
    Copy link
    Member Author

    @1st1 1st1 commented Jan 25, 2018

    New changeset c9070d0 by Yury Selivanov in branch 'master':
    bpo-32662: Implement Server.start_serving() and Server.serve_forever() (bpo-5312)
    c9070d0

    @1st1 1st1 closed this as completed Jan 25, 2018
    @1st1
    Copy link
    Member Author

    @1st1 1st1 commented Jan 26, 2018

    New changeset 4112c5b by Yury Selivanov in branch 'master':
    bpo-32662: Try making test_asyncio.test_server more reliable (bpo-5338)
    4112c5b

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Jan 26, 2018

    test_asyncio.test_start_server_1() hangs on "s390x Debian 3.x" buildbot:
    http://buildbot.python.org/all/#builders/13/builds/579

    test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
    test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
    Exception in thread test-client:
    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
        self._prog(TestSocketWrapper(self._sock))
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
        with self.tcp_client(lambda sock: client(sock, addr)):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
        sock.connect(addr)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/threading.py", line 917, in _bootstrap_inner
        self.run()
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
        self._test._abort_socket_test(ex)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
        self.fail(ex)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 680, in fail
        raise self.failureException(msg)
    AssertionError: [Errno 111] Connection refused

    Timeout (0:15:00)!
    Thread 0x000003fffcdcd710 (most recent call first):
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/selectors.py", line 468 in select
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 1555 in _run_once
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 450 in run_forever
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 615 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 663 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1861 in _run_suite
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1951 in run_unittest
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 176 in runtest_inner
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 291 in rerun_failed_tests
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 540 in _main
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 510 in main
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 585 in main
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/main.py", line 2 in <module>
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
    File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 193 in _run_module_as_main
    test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Makefile:1090: recipe for target 'buildbottest' failed
    make: *** [buildbottest] Error 1
    program finished with exit code 2
    elapsedTime=2050.270461

    @vstinner vstinner reopened this Jan 26, 2018
    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Jan 26, 2018

    The test fails differently on s390x SLES 3.x:
    http://buildbot.python.org/all/#/builders/16/builds/576

    ...
    test_force_close (test.test_asyncio.test_selector_events.SelectorTransportTests) ... ok
    test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
    test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
    test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
        self._prog(TestSocketWrapper(self._sock))
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
        with self.tcp_client(lambda sock: client(sock, addr)):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
        sock.connect(addr)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/threading.py", line 917, in _bootstrap_inner
        self.run()
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
        self._test._abort_socket_test(ex)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
        self.fail(ex)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 680, in fail
        raise self.failureException(msg)
    AssertionError: [Errno 111] Connection refused
    
    /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:45: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 56340)>
      self.loop.run_until_complete(main_task)
    ERROR
    test_start_unix_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... ok
    test_start_tls_client_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
    test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
    test_start_tls_wrong_args (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
    test_start_tls_client_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/collections/__init__.py:404: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 42302)>
      result = _self._make(map(kwds.pop, field_names, _self))
    Unhandled error in exception handler
    context: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<BaseStartServer.test_start_server_1.<locals>.main() done, defined at /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:31> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x3fff979bbf8>()]>>}
    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 1459, in call_exception_handler
        self._exception_handler(self, context)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 22, in loop_exception_handler
        self.loop.default_exception_handler(context)
    AttributeError: 'NoneType' object has no attribute 'default_exception_handler'
    ok
    test_start_tls_server_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
    test_start_tls_wrong_args (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
    test_cancel_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) ... ok
    test_close_during_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) .
    ...

    ======================================================================
    ERROR: test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 45, in test_start_server_1
        self.loop.run_until_complete(main_task)
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 493, in run_until_complete
        raise RuntimeError('Event loop stopped before Future completed.')
    RuntimeError: Event loop stopped before Future completed.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Jan 26, 2018

    Just to be complete, the test also fails on x86 Gentoo Non-Debug with X 3.x:
    http://buildbot.python.org/all/#/builders/99/builds/564

    ...
    test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
    Traceback (most recent call last):
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 193, in run
        self._prog(TestSocketWrapper(self._sock))
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
        with self.tcp_client(lambda sock: client(sock, addr)):
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 19, in client
        sock.connect(addr)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/threading.py", line 917, in _bootstrap_inner
        self.run()
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 195, in run
        self._test._abort_socket_test(ex)
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
        self.fail(ex)
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 680, in fail
        raise self.failureException(msg)
    AssertionError: [Errno 111] Connection refused

    Timeout (0:15:00)!
    Thread 0xb7d69700 (most recent call first):
    File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/selectors.py", line 468 in select
    File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 1555 in _run_once
    File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 450 in run_forever
    File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
    File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
    ...

    @1st1
    Copy link
    Member Author

    @1st1 1st1 commented Jan 26, 2018

    I think these failures were on the previous version of the test; i've fixed it last night and it shouldn't timeout this way anymore.

    @1st1 1st1 closed this as completed May 29, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 expert-asyncio type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants