Test failures with Python 3.11 pre-release (3.11b3)

[musicinmybrain]

Describe the bug
Several tests fail when testing Python 3.11 beta 3 for Fedora Linux 37. The development version of Fedora will start to use a pre-release of Python 3.11 soon, and Fedora 37 will be released with Python 3.11 this fall.

Downstream bug report

To Reproduce
Steps to reproduce the behavior:

1. Check out current main.
2. Edit tox.ini to add a Python 3.11 environment:

diff --git a/tox.ini b/tox.ini
index 146ccad..7a0766d 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,5 +1,5 @@
 [tox]
-envlist=flake8,py37,py38,py39,py310,pypy3,docs
+envlist=flake8,py37,py38,py39,py310,py311,pypy3,docs
 skip_missing_interpreters=True
 
 [gh-actions]

3. Make sure the appropriate interpreter is available. (On Fedora Linux 36: sudo dnf install python3.11 python3.11-devel)
4. Run tox. (Or tox -e py311.)

Unfortunately, this isn’t quite enough to reproduce what we’re seeing, because you’ll be blocked by problems with dependencies that we’ve worked around downstream. Hopefully the fixes will soon be available in the upstream releases of those dependencies.

Expected behavior
All tests pass in all environments. (In my local environment, something breaks on pypy3 that I haven’t looked into, but that’s not relevant here.)

Logs

=================================== FAILURES ===================================
____________________ TestAsyncServer.test_connect_bad_poll _____________________

self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_bad_poll>
import_module = <MagicMock name='import_module' id='140656792811152'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656792810384'>

    @mock.patch('engineio.asyncio_socket.AsyncSocket')
    @mock.patch('importlib.import_module')
    def test_connect_bad_poll(self, import_module, AsyncSocket):
        a = self.get_async_mock()
        import_module.side_effect = [a]
        AsyncSocket.return_value = self._get_mock_socket()
        AsyncSocket.return_value.poll.side_effect = [exceptions.QueueEmpty]
        s = asyncio_server.AsyncServer()
        _run(s.handle_request('request'))
        assert a._async['make_response'].call_count == 1
>       assert a._async['make_response'].call_args[0][0] == '400 BAD REQUEST'
E       AssertionError: assert '200 OK' == '400 BAD REQUEST'
E         - 400 BAD REQUEST
E         + 200 OK

tests/asyncio/test_asyncio_server.py:291: AssertionError
_______________ TestAsyncServer.test_connect_transport_websocket _______________

self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_transport_websocket>
import_module = <MagicMock name='import_module' id='140656788334672'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656788328912'>

    @mock.patch('engineio.asyncio_socket.AsyncSocket')
    @mock.patch('importlib.import_module')
    def test_connect_transport_websocket(self, import_module, AsyncSocket):
        a = self.get_async_mock(
            {
                'REQUEST_METHOD': 'GET',
                'QUERY_STRING': 'transport=websocket',
                'HTTP_UPGRADE': 'websocket',
            }
        )
        import_module.side_effect = [a]
        AsyncSocket.return_value = self._get_mock_socket()
        s = asyncio_server.AsyncServer()
        s.generate_id = mock.MagicMock(return_value='123')
        # force socket to stay open, so that we can check it later
        AsyncSocket().closed = False
        _run(s.handle_request('request'))
        assert (
            s.sockets['123'].send.mock.call_args[0][0].packet_type
            == packet.OPEN
>       )
E       AttributeError: 'function' object has no attribute 'mock'

tests/asyncio/test_asyncio_server.py:313: AttributeError
___________ TestAsyncServer.test_connect_transport_websocket_closed ____________

self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_transport_websocket_closed>
import_module = <MagicMock name='import_module' id='140656791298256'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656791305040'>

    @mock.patch('engineio.asyncio_socket.AsyncSocket')
    @mock.patch('importlib.import_module')
    def test_connect_transport_websocket_closed(
        self, import_module, AsyncSocket
    ):
        a = self.get_async_mock(
            {
                'REQUEST_METHOD': 'GET',
                'QUERY_STRING': 'transport=websocket',
                'HTTP_UPGRADE': 'websocket'
            }
        )
        import_module.side_effect = [a]
        AsyncSocket.return_value = self._get_mock_socket()
        s = asyncio_server.AsyncServer()
        s.generate_id = mock.MagicMock(return_value='123')
    
        # this mock handler just closes the socket, as it would happen on a
        # real websocket exchange
        async def mock_handle(environ):
            s.sockets['123'].closed = True
    
        AsyncSocket().handle_get_request = mock_handle
        _run(s.handle_request('request'))
>       assert '123' not in s.sockets  # socket should close on its own
E       AssertionError: assert '123' not in {'123': <engineio.asyncio_socket.AsyncSocket object at 0x7fed364c6810>}
E        +  where {'123': <engineio.asyncio_socket.AsyncSocket object at 0x7fed364c6810>} = <engineio.asyncio_server.AsyncServer object at 0x7fed35ff0c90>.sockets

tests/asyncio/test_asyncio_server.py:361: AssertionError
_____________________ TestAsyncServer.test_disconnect_all ______________________

self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_disconnect_all>

    def test_disconnect_all(self):
        s = asyncio_server.AsyncServer()
        s.sockets['foo'] = mock_foo = self._get_mock_socket()
        s.sockets['bar'] = mock_bar = self._get_mock_socket()
>       _run(s.disconnect())

tests/asyncio/test_asyncio_server.py:157: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/asyncio/test_asyncio_server.py:33: in _run
    return asyncio.get_event_loop().run_until_complete(coro)
/usr/lib64/python3.11/asyncio/base_events.py:650: in run_until_complete
    return future.result()
../../BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_server.py:178: in disconnect
    await asyncio.wait([client.close()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fs = {<coroutine object AsyncMock.<locals>.mock_coro at 0x7fed361f0e50>, <coroutine object AsyncMock.<locals>.mock_coro at 0x7fed361f05e0>}

    async def wait(fs, *, timeout=None, return_when=ALL_COMPLETED):
        """Wait for the Futures or Tasks given by fs to complete.
    
        The fs iterable must not be empty.
    
        Coroutines will be wrapped in Tasks.
    
        Returns two sets of Future: (done, pending).
    
        Usage:
    
            done, pending = await asyncio.wait(fs)
    
        Note: This does not raise TimeoutError! Futures that aren't done
        when the timeout occurs are returned in the second set.
        """
        if futures.isfuture(fs) or coroutines.iscoroutine(fs):
            raise TypeError(f"expect a list of futures, not {type(fs).__name__}")
        if not fs:
            raise ValueError('Set of Tasks/Futures is empty.')
        if return_when not in (FIRST_COMPLETED, FIRST_EXCEPTION, ALL_COMPLETED):
            raise ValueError(f'Invalid return_when value: {return_when}')
    
        fs = set(fs)
    
        if any(coroutines.iscoroutine(f) for f in fs):
>           raise TypeError("Passing coroutines is forbidden, use tasks explicitly.")
E           TypeError: Passing coroutines is forbidden, use tasks explicitly.

/usr/lib64/python3.11/asyncio/tasks.py:424: TypeError
______________ TestAsyncServer.test_http_upgrade_case_insensitive ______________

self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_http_upgrade_case_insensitive>
import_module = <MagicMock name='import_module' id='140656789877840'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656789870800'>

    @mock.patch('engineio.asyncio_socket.AsyncSocket')
    @mock.patch('importlib.import_module')
    def test_http_upgrade_case_insensitive(self, import_module, AsyncSocket):
        a = self.get_async_mock(
            {
                'REQUEST_METHOD': 'GET',
                'QUERY_STRING': 'transport=websocket',
                'HTTP_UPGRADE': 'WebSocket',
            }
        )
        import_module.side_effect = [a]
        AsyncSocket.return_value = self._get_mock_socket()
        s = asyncio_server.AsyncServer()
        s.generate_id = mock.MagicMock(return_value='123')
        # force socket to stay open, so that we can check it later
        AsyncSocket().closed = False
        _run(s.handle_request('request'))
        assert (
            s.sockets['123'].send.mock.call_args[0][0].packet_type
            == packet.OPEN
>       )
E       AttributeError: 'function' object has no attribute 'mock'

tests/asyncio/test_asyncio_server.py:335: AttributeError
=============================== warnings summary ===============================
../../../../usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:6
  /usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    preserves_excinfo = (distutils.version.LooseVersion(greenlet.__version__)

../../../../usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:7
  /usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:7: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    >= distutils.version.LooseVersion('0.3.2'))

tests/asyncio/test_async_asgi.py: 46 warnings
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_async_asgi.py:23: DeprecationWarning: There is no current event loop
    return asyncio.get_event_loop().run_until_complete(coro)

tests/asyncio/test_async_tornado.py::TornadoTests::test_translate_request
tests/asyncio/test_async_tornado.py::TornadoTests::test_translate_request
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_async_tornado.py:16: DeprecationWarning: There is no current event loop
    return asyncio.get_event_loop().run_until_complete(coro)

tests/asyncio/test_asyncio_client.py: 83 warnings
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:33: DeprecationWarning: There is no current event loop
    return asyncio.get_event_loop().run_until_complete(coro)

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
  /builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_client.py:177: DeprecationWarning: There is no current event loop
    return asyncio.ensure_future(target(*args, **kwargs))

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:238: DeprecationWarning: There is no current event loop
    pending = asyncio.all_tasks(loop=asyncio.get_event_loop()) \

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:240: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(asyncio.wait(pending))

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_disconnect_polling_abort
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:212: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
    assert c not in client.connected_clients
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_disconnect_websocket_abort
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:228: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
    assert c not in client.connected_clients
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_signal_handler
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:1456: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(test())

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_coroutine_async
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:943: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_coroutine_async_error
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:971: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_function_async
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:930: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_function_async_error
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:957: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_connection_no_open_packet
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_connection_with_cookies
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_upgrade_no_pong
tests/common/test_client.py::TestClient::test_signal_handler
tests/common/test_client.py::TestClient::test_signal_handler
tests/common/test_client.py::TestClient::test_signal_handler
  /builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_client.py:200: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
  /builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_server.py:332: DeprecationWarning: There is no current event loop
    return asyncio.ensure_future(target(*args, **kwargs))

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1012: DeprecationWarning: There is no current event loop
    pending = asyncio.all_tasks(loop=asyncio.get_event_loop()) \

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1014: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(asyncio.wait(pending))

tests/asyncio/test_asyncio_server.py: 68 warnings
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:33: DeprecationWarning: There is no current event loop
    return asyncio.get_event_loop().run_until_complete(coro)

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_coroutine_async
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1094: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_coroutine_async_error
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1122: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_function_async
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1081: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_function_async_error
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1108: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(fut)

tests/asyncio/test_asyncio_socket.py: 50 warnings
  /builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_socket.py:28: DeprecationWarning: There is no current event loop
    return asyncio.get_event_loop().run_until_complete(coro)

tests/common/test_client.py::TestClient::test_read_loop_websocket_disconnected
  /usr/lib64/python3.11/inspect.py:3081: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
    arg_val = next(arg_vals)
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_bad_poll
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket_closed
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_disconnect_all
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_http_upgrade_case_insensitive
================= 5 failed, 475 passed, 275 warnings in 5.66s ==================

Additional context

I’m not sure it will be trivial for me to prepare a PR to fix these, but I will be happy to do any experiments that might be helpful, and to test any candidate fixes.

[miguelgrinberg]

I will look into these failures if they continue to exist when 3.11 is officially out.

[musicinmybrain]

I will look into these failures if they continue to exist when 3.11 is officially out.

I appreciate that. I hope it will be possible to investigate these a little earlier, though. By the time Python 3.11 final is released (2022-10-03), Fedora Linux 37 will be entering final freeze, and (if nothing is fixed before then) the python-engineio package and its dependent packages will be broken in the final release.

Would you consider revisiting this, say, once Python 3.11 is at the Release Candidate stage (scheduled 2022-08-01)?

[miguelgrinberg]

@musicinmybrain Would you be interested in debugging these problems on your side? I'm open to a PR if you figure out what's wrong in time.

[musicinmybrain]

@musicinmybrain Would you be interested in debugging these problems on your side? I'm open to a PR if you figure out what's wrong in time.

Yes, I will definitely spend some time studying it in the interim.

[musicinmybrain]

I can now reproduce this in a virtualenv, at least.

$ python3.11 -m venv _env
$ . _env/bin/activate
(_env) $ pip install --upgrade pip setuptools wheel
(_env) $ AIOHTTP_NO_EXTENSIONS=1 FROZENLIST_NO_EXTENSIONS=1 YARL_NO_EXTENSIONS=1 pip install -e .[client,asyncio_client] pytest tornado
(_env) $ pytest --ignore=tests/common/test_async_eventlet.py -k 'not test_async_mode_eventlet'

I get two WebSocket-related failures not reported above, and the process hangs and does not respond to ^C after printing the test summary, but I am at least able to reproduce the four remaining failures described in this report without relying on anything specific to Fedora Linux.

[musicinmybrain]

This is just a status update. I haven’t spent any more time trying to understand the remaining errors.

---

Now

AIOHTTP_NO_EXTENSIONS=1 FROZENLIST_NO_EXTENSIONS=1 YARL_NO_EXTENSIONS=1 pip install -e .[client,asyncio_client] pytest tornado

can simply be

pip install -e .[client,asyncio_client] pytest tornado

---

The result of

pytest --ignore=tests/common/test_async_eventlet.py -k 'not test_async_mode_eventlet'

has not changed since my previous comment.

---

Fedora Rawhide is now using Python 3.11b3, but Python 3.11 in Fedora 37 is at risk due to possible CPython upstream schedule slip.

[musicinmybrain]

No change with 3.11b4, which just came out.

[musicinmybrain]

No change with Python 3.11b5 and python-engineio 4.3.4.

Incidentally, it looks like Python 3.11 in Fedora 37 is likely going to stay on track.

We are just skipping these tests in Fedora Rawhide/37 for the time being in order to unblock dependent packages. We are guessing that these failures likely reflect a problem with mocking rather than a problem with the library’s operation, but we don’t understand what’s happening well enough to fix it.

[miguelgrinberg]

I can confirm that this is an issue with the mocking that I'm doing, which causes some tests to fail. There are no indications that the actual package is failing under 3.11.

I haven't been able to determine what the problem is yet. Something within asyncio and/or the unittest.mock package must have changed in a way that only in four tests in this package a mock that is inserted on the AsyncSocket class is not applied, causing the tests to run with the real object instead of the mocked one, thus failing.

FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_bad_poll - AssertionError: assert '200 OK' == '400 BAD REQUEST'
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket - AttributeError: 'function' object has no attribute 'mock'
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket_closed - AssertionError: assert '123' not in {'123': <engineio.asyncio_socket.AsyncSocket object...
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_http_upgrade_case_insensitive - AttributeError: 'function' object has no attribute 'mock'

[musicinmybrain]

Thank you for the update!

[miguelgrinberg]

This patch to unittest.mock caused the failures. The mock patching did not import anything in 3.10 and older. In 3.11 it does import something, so mocking fails when the import_module() function has also been mocked and cannot be used.

For now I'm going to reverse the order of my mocks, so that import_module() is done last. Longer term I should probably figure out a way to mock without messing with such a core function of Python. :)