Statistics for all available channels?

[approxit]

Having global_statistics() and channel_statistics() is actually the only way to figure out what's going on with channels performance with asgi_redis. The problem with channels_statistics() - it's hard to get statistics about response channels, that can be full too.

Playing with performance tuning and a little bigger scale (~200 websocket connections, 6 daphnes, 12 workers) is it very hard to measure lack of daphnes or lack of workers.

Is there any way to get statistics about all available channels?

[proofit404]

I don't think there are any special statistic methods for reply channels.

Probably the most useful way will be to run stress test against a real project. Compare how many time was spent inside consumer against total message processing time must give you proper insight.

[andrewgodwin]

channel_statistics is particularly unhelpful for response channels, I agree, but we can't list all channels as that in itself might be very slow, especially if things are sharded. If you could give examples of the sort of things you might want to know, we can think about ways to get them semi-efficiently.

[approxit]

My wet dreams would be have some kind of channels htop thing, to see actually everything about performance. Interface servers and their load, workers and their load, channels and their load. But we know how it is with dreams...

Right now I'm receiving tons of channel full messages on "daphne.response" and I realized that I'm totally blind on outgoing messages from channels server.

Current parameters like in channel_statistics are totally enough for making load charts, or trigger automatic deployment to spawn more traffic lifting, etc. I would love to have a little more sources.

I want to archive ability to know when I'm close to my capacity and act before error raises and messages are dropped. Anything that can tell me "that daphne instance can't handle more load" or "that websocket connection is way too heavy" would do the job perfectly.

[proofit404]

I have plans to work on some kind of monitoring tool. I don't know what form it will have at the end. For now, I just have few thoughts.

Probably it will be a collection of predefined alerts for prometheus.io

Maybe this issue can be a sophisticated place to start this discussion.

[devenbhooshan]

@approxit 👍
Some of the alerts should be triggered when

1. Average queuing time of the messages in certain channel is greater than X ms(during some interval)
2. Y% (eg: 90%) of the channel capacity reached
3. Average time taken by the workers listening to a particular channel is greater than Z ms.

[proofit404]

@devenbhooshan Can you clarify some conditions for me?

2. Is to broad, some channel layers don't expose current message number in the channel. This alert is nice to have, but maybe layer specific.
3. What is the purpose of this alert?

[andrewgodwin]

Statistics have been removed from channels_redis in version 2.0 as it's no longer used for sending HTTP request/response data and it's meant to be lighter-weight overall (plus the Redis key structure is now a lot simpler and easier to query with external scripts), so closing this.