Topic specific statistics

[skinkie]

In threaddata.cpp some "$SYS" topics are currently defined. In your opinion:

1. Should the server be able to provide throughput per topic so monitoring it becomes possible?
2. Should a client aggregate these statistics, at the cost of receiving all information?

[halfgaar]

Per topic? That's going to be a lot of overhead for the system, and you won't be able to collect the stats. If there have been a million topics and you subscribe to the hypothetical path $SYS/pertopic/#, you'd need to get a million topics. This is not going to work, from FlashMQ's side, as well as clients'. Most clients are not able to receive more than a few thousand topics at once.

Using a client to count is an option. If you have a very high message volume you could even use shared subscriptions to spread it out over multiple clients.

Another option is writing a plugin. There are a myriad of ways to implement it, should you chose that path.

[skinkie]

When I am interested in the number of retained messages or the throughput, it seems that a client cannot implement this unless all data is received as well, while metadata is actually required. So the server would be the only place that would have this information available without redistributing the contents. There are a lot of ways to implement this, I just wanted to what you would consider core functionality.

I think something like the following structure could trigger the counting $SYS/statistics/throughput/topic/# which would count everything under that. But $SYS/statistics/throughput/topic would only aggregate that topic, and $SYS/statistics/count/topic/# query all retained messages.

In order for this to work dynamically the plugin should have access to the subscription table, and only start processing then.

[halfgaar]

I think it's too specific as core functionality, with too much impact on performance and chance for it to grow out of control.

In the plugin, you can hook into subscribe actions to do what you describe, or just program it out so that it always does the counting you require.

One example of doing it, is to send an UDP datagram from flashmq_plugin_acl_check(...) on 'write', with the topic name. This way it's completely asynchronous and you can implement the counting in whatever way you want.

Statsd works like that too. That's something I'd consider implementing, but I know from experience that one wants your tag volume low otherwise the data becomes too big to handle, so tagging with topic name is probably a bridge too far.