Is there a test client?

[jacopofar]

Hello and thanks for the library!

I'm using it with Starlette and trying to implement some integration test. Is there a test client for socketio similar to the one they provide for the basic HTTP/websocket (here), or examples about how to implement such a test?

[miguelgrinberg]

There is currently no test client. I have one for the Flask integration with this package, it would be nice to have something similar that is generic.

The best approximation is to use a real server with a real client, both possibly running within the same process.

[jacopofar]

Thanks. I tried to implement the test in such a way, but cannot manage to have the ASGI server running in a separate thread or process while the test client connects to it.

My test looks like this:

@pytest.mark.asyncio
async def test_websocket():
    config = uvicorn.Config(app.create_app(), host='localhost', port=8000)
    server = uvicorn.Server(config)
    loop = asyncio.get_running_loop()

    serve_coroutine = await server.serve()
    executor = concurrent.futures.ProcessPoolExecutor(
        max_workers=5,
    )
    loop.run_in_executor(executor, serve_coroutine)

    # this line is reached only when I press Ctrl-C and kill the server

    async def connect_to_ws():
        sio = socketio.Client()
        sio.connect('http://localhost:8000')
        # here would go assertions on the socket responses

    k = await connect_to_ws()
    loop.run_in_executor(executor, k)

the app.create_app function is:

def create_app():

    app = Starlette(debug=True)
    app.mount('/', StaticFiles(directory='static'), name='static')

    sio = socketio.AsyncServer(async_mode='asgi')
    extended_app = socketio.ASGIApp(sio, app)

    # here define HTTP and Socketio handlers

    return extended_app

the basic idea is to start the complete server and just connect to it, I assumed I could run the server and the test client in the same event loop but apparently when I run the server (that is indeed started and I can reach with the browser) it blocks the test code. Only when I use Ctrl-C to stop it the server is killed and the rest of the test runs but of course it doesn't find the server.

Probably I'm missing something essential here, I expected the server and the test client to run concurrently on the same event loop without need for multithreading or multiprocessing.

[miguelgrinberg]

In your example you are using a process executor, so you are in fact using multiprocessing there. I think this can be done in a much simpler way. Here is a rough attempt that appears to be work well:

import asyncio
import socketio

sio = socketio.AsyncServer(async_mode='asgi', monitor_clients=False)
app = socketio.ASGIApp(sio)

def start_server():
    import asyncio
    from uvicorn import Config, Server
    config = Config(app, host='127.0.0.1', port=5000)
    server = Server(config=config)
    config.setup_event_loop()
    loop = asyncio.get_event_loop()
    server_task = server.serve()
    asyncio.ensure_future(server_task)
    return server_task

async def run_client():
    client = socketio.AsyncClient()
    await client.connect('http://localhost:5000')
    await asyncio.sleep(5)
    await client.disconnect()

start_server()
loop = asyncio.get_event_loop()
loop.run_until_complete(run_client())

Hopefully this will get you started.

[jacopofar]

Thanks a lot! Indeed from this example I was able to make it work :)

For whoever will encounter the problem in the future, in case someone in the future is interested this is my implementation:

The app:

def create_app():

    app = Starlette(debug=True)
    app.mount('/', StaticFiles(directory='static'), name='static')

    sio = socketio.AsyncServer(async_mode='asgi')
    extended_app = socketio.ASGIApp(sio, app)

    @sio.on('double')
    async def double(sid, data):
        logging.info(f"doubling for {sid}")
        return 'DOUBLED:' + data * 2
   # here add HTTP and WS handlers...
   return extended_app

The test, based on pytest-asyncio, uses an async ficture to start and stop the server has this structure:

import asyncio

import socketio
import uvicorn
import pytest

from myapp import app

def get_server():
    config = uvicorn.Config(app.create_app(), host='localhost', port=8000)
    server = uvicorn.Server(config=config)
    config.setup_event_loop()
    return server


@pytest.fixture
async def async_get_server():
    server = get_server()
    server_task = server.serve()
    asyncio.ensure_future(server_task)

@pytest.mark.asyncio
async def test_websocket(async_get_server):
    client = socketio.AsyncClient()
    await client.connect('http://localhost:8000')
    result = await client.call('double', 'hello')
    assert result == 'DOUBLED:hellohello'
    await client.disconnect()

This work although it produces a lot of warnings (I suspect some problem with the logs).

A weird thing I noticed is that to run this it is required to install aiohttp, if not I get timeout errors. Could it be worth to raise an explicit error in this case? I can try and do a PR if it's fine for you

[miguelgrinberg]

The aiohttp package provides the WebSocket client, without it the connection stays as long-polling. Not sure why the timeouts without it however, I'll have to test that.

[nbanmp]

I'm at a point where having a test client with socketio would be really helpful for me too. Is there any update on the progress of this?

[miguelgrinberg]

@nbanmp I am not currently working on a test client. The main reason is that there is a real Python client now. Is there anything that prevents you from using the real client against your real server for tests?

[nbanmp]

Thanks for the update.

Running the real client against the real server has some issues, the main one for me is that it is more difficult to run multiple tests asynchronously. But also important is that, I would like my unit tests to be as independent as possible, and I was expecting to run the actual server for integration testing.

[miguelgrinberg]

In any case, the test client that exists in the Flask-SocketIO package is very limited, if/when I get to do a generic test client it would be based on the real client talking to the real server. It would make it easier to start/stop the server for each test, but other than that I expect it will be the real thing, not a fake.

[databasedav]

There should to be a way to gracefully shutdown the server given the setup above. This would need to cancel the _service_task and disconnect the remaining connect clients. I can make this contribution but need some guidance on a few things, particularly how the _service_task is started from a socketio.AsyncServer; I can find the task being started in engineio.AsyncServer but not in the former.

[miguelgrinberg]

@databasedav The service task does not need to run when testing. A testing set up can be made by subclassing the Server and Client classes (and their asyncio counterparts) to re-implement the networking parts through direct calls.

[databasedav]

@miguelgrinberg I agree; I was just talking in the context of using a live server like discussed above

[miguelgrinberg]

@databasedav start your server with monitor_clients=False to disable the service task. I'll actually add that.

[Korijn]

In your example you are using a process executor, so you are in fact using multiprocessing there. I think this can be done in a much simpler way. Here is a rough attempt that appears to be work well:

...

Hopefully this will get you started.

This definitely did the trick! For future readers, I also had to do the following:

• Install aiohttp in order for the client to work
• Set start_service_task = False on the server engineio object

If you also want to work with HTTP requests in the same client session, use client.eio._send_request or client.eio.http, that way things like cookies will be shared

I also used the following to shutdown after the test:

server.should_exit = True
loop.run_until_complete(server_task)

I do still wonder if it's possible to set this up directly on ASGI level, instead of actually binding to ports and hostnames...

[erny]

Ok, here goes a complete example, using FastAPI as the primary ASGI app and socketio as the secondary one. The chat server echoes the message to all clients.

• UPDATE 1: includes @Korijn's improvements.
• UPDATE 2: Now, we use a future to wait for the result
• UPDATE 3: @Korijn's improvements (timed wait for server startup) has been replaced with a asyncio.Event sync mechanism

src/app/main.py:

import os
import socketio

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()
path = os.path.dirname(__file__)
app.mount("/static", StaticFiles(directory=os.path.join(path, "static")), name="static")

sio = socketio.AsyncServer(async_mode='asgi')
app.mount('/sio', socketio.ASGIApp(sio))  # socketio adds automatically /socket.io/ to the URL.


@sio.on('connect')
def sio_connect(sid, environ):
    """Track user connection"""
    print('A user connected')


@sio.on('disconnect')
def sio_disconnect(sid):
    """Track user disconnection"""
    print('User disconnected')


@sio.on('chat message')
async def chat_message(sid, msg):
    """Receive a chat message and send to all clients"""
    print(f"Server received: {msg}")
    await sio.emit('chat message', msg)

src/app/tests/test_chat.py:

from typing import List, Optional
# stdlib imports
import asyncio

# 3rd party imports
import pytest
import socketio
import uvicorn

# FastAPI imports
from fastapi import FastAPI

# project imports
from .. import main

PORT = 8000

# deactivate monitoring task in python-socketio to avoid errores during shutdown
main.sio.eio.start_service_task = False


class UvicornTestServer(uvicorn.Server):
    """Uvicorn test server

    Usage:
        @pytest.fixture
        async def start_stop_server():
            server = UvicornTestServer()
            await server.up()
            yield
            await server.down()
    """

    def __init__(self, app: FastAPI = main.app, host: str = '127.0.0.1', port: int = PORT):
        """Create a Uvicorn test server

        Args:
            app (FastAPI, optional): the FastAPI app. Defaults to main.app.
            host (str, optional): the host ip. Defaults to '127.0.0.1'.
            port (int, optional): the port. Defaults to PORT.
        """
        self._startup_done = asyncio.Event()
        super().__init__(config=uvicorn.Config(app, host=host, port=port))

    async def startup(self, sockets: Optional[List] = None) -> None:
        """Override uvicorn startup"""
        await super().startup(sockets=sockets)
        self.config.setup_event_loop()
        self._startup_done.set()

    async def up(self) -> None:
        """Start up server asynchronously"""
        self._serve_task = asyncio.create_task(self.serve())
        await self._startup_done.wait()

    async def down(self) -> None:
        """Shut down server asynchronously"""
        self.should_exit = True
        await self._serve_task


@pytest.fixture
async def startup_and_shutdown_server():
    """Start server as test fixture and tear down after test"""
    server = UvicornTestServer()
    await server.up()
    yield
    await server.down()


@pytest.mark.asyncio
async def test_chat_simple(startup_and_shutdown_server):
    """A simple websocket test"""

    sio = socketio.AsyncClient()
    future = asyncio.get_running_loop().create_future()

    @sio.on('chat message')
    def on_message_received(data):
        print(f"Client received: {data}")
        # set the result
        future.set_result(data)

    message = 'Hello!'
    await sio.connect(f'http://localhost:{PORT}', socketio_path='/sio/socket.io/')
    print(f"Client sends: {message}")
    await sio.emit('chat message', message)
    # wait for the result to be set (avoid waiting forever)
    await asyncio.wait_for(future, timeout=1.0)
    await sio.disconnect()
    assert future.result() == message

Here goes a test run:
$ pytest -s src/app/tests/test_chat.py

=============================================================== test session starts ================================================================
platform darwin -- Python 3.7.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/erevilla/Documents/proyectos/next/product/playground
plugins: cov-2.10.1, asyncio-0.14.0
collected 1 item                                                                                                                                  

src/app/tests/test_chat.py INFO:     Started server process [2927]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
A user connected
INFO:     127.0.0.1:63597 - "GET /sio/socket.io/?transport=polling&EIO=3&t=1603207299.770268 HTTP/1.1" 200 OK
INFO:     ('127.0.0.1', 63597) - "WebSocket /sio/socket.io/" [accepted]
Client sends: Hello!
Server received and sends to all clients: Hello!
Client received: Hello!
.User disconnected
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [2927]


================================================================ 1 passed in 1.02s =================================================================

Notes:

• Start the test without -s to hide the server output.

[Korijn]

@erny some suggestions for improvement over the hardcoded sleeps (untested), which should make your test suite faster overall:

async def wait_ready(server, interval=0.05, max_wait=5):
    i = 0
    while not server.started:
        await asyncio.sleep(interval)
        i += interval
        if i > max_wait:
            raise RuntimeError()

@pytest.fixture
async def async_get_server():
    """Start server as test fixture and tear down after test"""
    server = get_server()
    serve_task = asyncio.create_task(server.serve())
    await wait_ready(server)   # wait for the server to startup
    yield 1
    # teardown code
    server.should_exit = True
    await serve_task()  # allow server run tasks before shut down

[miguelgrinberg]

Awesome, thanks @erny! Given that this appears to be figured out, I'm going to close this issue.

[erny]

@Korijn I included your improvements in the updated example. Thank you very much.

[erny]

@Korijn, @miguelgrinberg , I was not able to remove the sio.sleep(0.1) after sio.emit. Is there any alternative?

[Korijn]

@Korijn, @miguelgrinberg , I was not able to remove the sio.sleep(0.1) after sio.emit. Is there any alternative?

You would need to wait in a loop for result.message_received to become True, very similar to how wait_ready is defined. You could pass the condition to wait for as an argument to make it reusable.

async def wait_ready(condition, interval=0.05, max_wait=5):
    i = 0
    while not condition():
        await asyncio.sleep(interval)
        i += interval
        if i > max_wait:
            raise RuntimeError()

Usage examples:

await wait_ready(lambda: server.started)
await wait_ready(lambda: result.message_received)

Also I guess you could still lower the interval quite a bit, like 0.001 or even lower maybe.

[erny]

I was thinking about defining the result as future, something a bit more elegant, but I'm not sure if I'll be able to do it. Regards El mar., 20 oct. 2020 21:40, Korijn van Golen <notifications@github.com> escribió:

…

@Korijn <https://github.com/Korijn>, @miguelgrinberg <https://github.com/miguelgrinberg> , I was not able to remove the sio.sleep(0.1) after sio.emit. Is there any alternative? You would need to wait in a loop for result.message_received to become True, very similar to how wait_ready is defined. You could pass the condition to wait for as an argument to make it reusable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#332 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAS24ELXQN6SWEQOBIQDLLSLXRUFANCNFSM4IKG6JCQ> .

[Korijn]

An idea would be to define a test server app with a catch-all event handler which writes all messages it receives to a list, and a helper to wait for a new message to come in which could also return that new message.

[SeeringPhil]

Would it be possible for one of you guys that have been able to make this work to list the versions of the libraries/dependencies you're using?

I attempted to replicate what's discussed here (#332 (comment)) using windows, but I keep having issues and I'm wondering if it's related to my python version or dependencies's.

Thanks !

Update:
I was able to make it work using linux (wsl2), but I'm experimenting an issue where the test takes a whole minute to "complete". In fact, I added a test case where I only changed the message to another word, and it takes exactly 2 minutes before completion. Every test stays stuck at

INFO:     Shutting down
INFO:     Waiting for background tasks to complete. (CTRL+C to force quit)

for a whole minute before going on with the next one.

[Korijn]

Put your code up somewhere so we can have a look 👍

[erny]

Hi.

Sorry for the late answer.

Would it be possible for one of you guys that have been able to make this work to list the versions of the libraries/dependencies you're using?

Of course, here we go:

python 3.7.9 (using a local pyenv installed instance and also using docker image python:3.7.9-slim)
uvicorn==0.12.2
  - click [required: ==7.*, installed: 7.1.2]
  - h11 [required: >=0.8, installed: 0.11.0]
  - typing-extensions [required: Any, installed: 3.7.4.3]
uvloop==0.14.0
aiofiles==0.5.0
aiohttp==3.6.3
  - async-timeout [required: >=3.0,<4.0, installed: 3.0.1]
  - attrs [required: >=17.3.0, installed: 20.2.0]
  - chardet [required: >=2.0,<4.0, installed: 3.0.4]
  - multidict [required: >=4.5,<5.0, installed: 4.7.6]
  - yarl [required: >=1.0,<1.6.0, installed: 1.5.1]
    - idna [required: >=2.0, installed: 2.10]
    - multidict [required: >=4.0, installed: 4.7.6]
    - typing-extensions [required: >=3.7.4, installed: 3.7.4.3]
fastapi==0.61.1
  - pydantic [required: >=1.0.0,<2.0.0, installed: 1.6.1]
  - starlette [required: ==0.13.6, installed: 0.13.6]
flake8==3.8.4
  - importlib-metadata [required: Any, installed: 2.0.0]
    - zipp [required: >=0.5, installed: 3.3.1]
  - mccabe [required: >=0.6.0,<0.7.0, installed: 0.6.1]
  - pycodestyle [required: >=2.6.0a1,<2.7.0, installed: 2.6.0]
  - pyflakes [required: >=2.2.0,<2.3.0, installed: 2.2.0]
httptools==0.1.1
pytest-asyncio==0.14.0
  - pytest [required: >=5.4.0, installed: 6.1.1]
    - attrs [required: >=17.4.0, installed: 20.2.0]
    - importlib-metadata [required: >=0.12, installed: 2.0.0]
      - zipp [required: >=0.5, installed: 3.3.1]
    - iniconfig [required: Any, installed: 1.1.1]
    - packaging [required: Any, installed: 20.4]
      - pyparsing [required: >=2.0.2, installed: 2.4.7]
      - six [required: Any, installed: 1.15.0]
    - pluggy [required: >=0.12,<1.0, installed: 0.13.1]
      - importlib-metadata [required: >=0.12, installed: 2.0.0]
        - zipp [required: >=0.5, installed: 3.3.1]
    - py [required: >=1.8.2, installed: 1.9.0]
    - toml [required: Any, installed: 0.10.1]
pytest-cov==2.10.1
  - coverage [required: >=4.4, installed: 5.3]
  - pytest [required: >=4.6, installed: 6.1.1]
    - attrs [required: >=17.4.0, installed: 20.2.0]
    - importlib-metadata [required: >=0.12, installed: 2.0.0]
      - zipp [required: >=0.5, installed: 3.3.1]
    - iniconfig [required: Any, installed: 1.1.1]
    - packaging [required: Any, installed: 20.4]
      - pyparsing [required: >=2.0.2, installed: 2.4.7]
      - six [required: Any, installed: 1.15.0]
    - pluggy [required: >=0.12,<1.0, installed: 0.13.1]
      - importlib-metadata [required: >=0.12, installed: 2.0.0]
        - zipp [required: >=0.5, installed: 3.3.1]
    - py [required: >=1.8.2, installed: 1.9.0]
    - toml [required: Any, installed: 0.10.1]
python-dotenv==0.14.0
python-socketio==4.6.0
  - python-engineio [required: >=3.13.0, installed: 3.13.2]
    - six [required: >=1.9.0, installed: 1.15.0]
  - six [required: >=1.9.0, installed: 1.15.0]
PyYAML==5.3.1
watchgod==0.6
websockets==8.1

(I just put the uvicorn deps on the top and skipped the mypy and jupyterlab dependencies which are very long...)

I attempted to replicate what's discussed here (#332 (comment)) using windows, but I keep having issues and I'm wondering if it's related to my python version or dependencies's.
May be...

Thanks !

Update:
I was able to make it work using linux (wsl2), but I'm experimenting an issue where the test takes a whole minute to "complete". In fact, I added a test case where I only changed the message to another word, and it takes exactly 2 minutes before completion. Every test stays stuck at

INFO:     Shutting down
INFO:     Waiting for background tasks to complete. (CTRL+C to force quit)
```0

for a whole minute before going on with the next one.

I have no "Waiting for background tasks to complete." message. Running pytest -s I get:

$ pytest -s
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.7.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/erevilla/Documents/proyectos/next/product/playground/src
plugins: cov-2.10.1, asyncio-0.14.0
collected 14 items                                                                                                                                                                                       

app/tests/test_chat.py INFO:     Started server process [33326]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
A user connected
INFO:     127.0.0.1:51226 - "GET /sio/socket.io/?transport=polling&EIO=3&t=1605442406.685194 HTTP/1.1" 200 OK
INFO:     ('127.0.0.1', 51226) - "WebSocket /sio/socket.io/" [accepted]
Client sends: Hello!
Server received and sends to all clients: Hello!
Client received: Hello!
.User disconnected
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [33326]
Chat page: /Users/erevilla/Documents/proyectos/next/product/playground/src/app/tests/../chat.html
.INFO:     Started server process [33326]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:51227 - "GET /chat HTTP/1.1" 200 OK
.INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [33326]

Searching inside the uvicorn source code we get in its main.py: 

        # Wait for existing tasks to complete.
        if self.server_state.tasks and not self.force_exit:
            msg = "Waiting for background tasks to complete. (CTRL+C to force quit)"
            logger.info(msg)
            while self.server_state.tasks and not self.force_exit:
                await asyncio.sleep(0.1)

It seems that I don't have a background task, but you do. What version of uvicorn are you using? Can you try to additionally set force_exit = True.