Daphne is slowly leaking memory via channels-redis

[ryanpetrello]

see: django/channels#1181 (comment)

[ryanpetrello]

In addition to the general leak described in Daphne, I've found another way to get Daphne to grow memory in an unbounded way.

Internally, as channels_redis consumes messages from Redis, it stores per-local-channel copies in an in-memory receive buffer for other consumers on the same topic. In this way, if multiple consumers are subscribed to the same topic (i.e., the stdout for a job, or the global websocket broadcast topic), they can each receive a copy.

Unfortunately, Daphne will continually grow this per-channel buffer in an unbounded way even if nothing seems to be reading from the other end (i.e., if the connection is closed, or if it's just reading too slowly).

To illustrate why this can become a problem:

1. Install a clustered AWX with 5 or more nodes.
2. Intentionally disrupt eth0 Node B, so that broadcasting from other nodes isn't read quickly, causing the buffer on Node A to fill (and never empty, because the read side on Node B can't keep up):

~ tc qdisc add dev eth0 root netem delay 500ms loss 50% 

3. Run some playbook that generates a high volume, constant stream of event data on Node A. Use instance groups to ensure that the playbook runs on Node A.
4. Note that Daphne's RSS on Node A will slowly grow in an unbounded way because the receiver buffer for the broadcast channel is filling and not being emptied.
5. Go eat a sandwich and come back 30 minutes later.

This is a fairy close approximation of the bug outlined at django/channels_redis#384 as it might affect AWX's busiest channel, the websocket backplane we use for broadcasting events to peers in a cluster. The messages in my testing are fairly small, so it takes awhile for memory to grow, but you could imagine that large messages (like lots of fact collection) would cause quicker memory growth.

related: django/channels_redis#384

[kdelee]

@ryanpetrello is this resolved via #8094 ?

[ryanpetrello]

Yes. Thanks, @kdelee.

[kdelee]

Upstream patch is merged and released and patch as been verified in production by users experiencing the bug. We've bumped the versions we depend on to use fix, closing.