Increasing number of asyncio tasks causing memory leak

[levesquejf]

I have a Websockets client that is always running. Once in a while, the client is disconnected from the server. It’s either because of network issues between the client and the server or because it is connected for too long and the client just reconnects to balance the charge on the servers. The client automatically reconnects to the server. I am using the websockets.connect() context to build the connection.

After a few weeks, the number of asyncio tasks is very high. When I list the tasks with asyncio.Task.all_tasks(), I see hundreds of websocket finished tasks:

• WebSocketCommonProtocol.close_connection()
• WebSocketCommonProtocol.transfer_data()

I am running Python 3.6.9 and Websockets 8.1. Any idea what could be the cause and how to troubleshoot further?

[aaugustin]

Are the tasks still there after import gc; gc.collect()?

Is there a chance that something in your program (including asyncio's debug mode) keeps a reference to previous client connections, preventing them from being garbage collected?

[levesquejf]

Running gc.collect() doesn't help. It removes some of the old tasks but not all.

I removed some logs that were referring to the connections and I upgraded to 3.7.5. I will deploy that version and let you know in a few days if it's better or not.

[levesquejf]

@aaugustin It's not better with my last changes. Any advice on how to troubleshoot?

[levesquejf]

The issue is only present with version >= 8.0 and when the code is compiled with Nuitka (see Nuitka/#588). I don't have the issue with version 7.0 so I will downgrade to that version for now. Since it's always running fine with native python, it's probably a bug with Nuitka. However, @aaugustin, if you have any info on what changed from 7.0 to 8.0 that can help the Nuitka team pinpoint the issue, I am sure they would appreciate.

A full reproducer is available at https://github.com/levesquejf/nuitka-websockets-bug.

[aaugustin]

I reviewed the changelog but I'm not spotting anything suspicious, nor do I remember changes around task management (termination / cancellation), sorry.

websockets doesn't get a lot of activity (137 commits between 7.0 and 8.0) and master is kept stable, so it should be fairly easy to bisect the regression with your reproducer. Checkout the repository, pip install -e, and bisect.