Load Testing Notes

[dennyabrain]

I'm writing some tests to understand how much load the backend server can handle. Will use this place to discuss preliminary results and findings. The intended outcome is a script that can load test a server and provide a csv file that can be analyzed to understand latency and performance decay as load increases.

[dennyabrain]

i see in main_loop/websocket.py, in the WebSocketRunner class, there is a mention of max_games, does this impose some artificial limit ? @RishavT

[dennyabrain]

I have noticed that my script can create 10 rooms sequentially and players inside those rooms can join simultaneously. But when I try to create 2 rooms simultaneously, the backend server seems to gets chocked.

Then when i hit ctrl+c on the server process, i see this stacktrace

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.9/site-packages/flask_socketio/__init__.py", line 824, in _handle_event
    ret = handler(auth)
  File "/app/main_loop/websocket.py", line 513, in connect
    with thread_lock:
KeyboardInterrupt

after which point the create_room and join_room events are successfully handled. I am not sure whats happening here but it looks like two simultaneous connections somehow "lock" the backend server.

[dennyabrain]

I can confirm that if i add an artificial sleep of random amount between the multiple connect calls, the server is able to handle creation and joining of 10 rooms. So something about simultaneous connect calls is locking the server.

[dennyabrain]

I think i've satisfactorily proved that this issue doesn't pertain to anything peculiar about our game's backend logic OR python (:P)
I created a barebones implementation of a socketio server in javascript and face the same problem.
I have described the problem in detail along with barebones implementation on stackoverflow and also on their slack channel.

Lets see if I hear anything from them.

[dennyabrain]

For now, I am proceeding with the assumption that socketio takes its time in establishing a connection with the server. We might not be able to do much here. The good thing is that the way the frontend is coded, a connect event is triggered the moment someone lands on the homepage or the room page, so even before they create a room, the connection will most likely have been established.

The server will choke if lets say 1000 people hit refresh on the homepage. I think having a multi node cluster is the only way to fix that.
I've made an issue for this - tattle-made/viral-spiral-backend#68