[RFE] read/write latency by table name

[dyasny]

Please make sure that this is a feature request.

System information

• Scylla version (you are using): 4.x/ 2019.1.x
• Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.
The customer would like to be able to monitor read and write latencies (avg/95/99) on a per table basis

Who will benefit with this feature?
This is a customer RFE from the field

Any Other info.
Ent. ticket 1358

[amnonh]

Data per table is not exposed by default and is not part of the monitoring stack

[slivne]

Amnon We do have metrics per CF if the user enables it - the issue is the amount of data it generates (especially if the user has a lot of data). If the user enables those metrics - they will have the data they are requesting.

…

On Tue, Jul 21, 2020 at 9:56 AM Amnon Heiman ***@***.***> wrote: Data per table is not exposed by default and is not part of the monitoring stack — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#998 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2OCCGT7B6R255R2X3RAJDR4U32PANCNFSM4PC7UE3A> .

[nyh]

Here is an excerpt from scylla.git's docs/metrics.md which tries to describe the current situation:

Most of Scylla's metrics are global (in each shard). Scylla also supports per-table metrics, which are maintained separately for each table in the database.

On a deployment with a large number of tables, this can result in a very large number of metrics at each time, and overwhelm Scylla's HTTP server and/or the Prometheus server collecting these metrics. For this reason, the per-table metrics are currently disabled by default:

To enable this flag and the per-table metrics, you can pass the parameters--enable-keyspace-column-family-metrics 1 in the Scylla command line, or set this parameter in Scylla's configuration file.

We are planning to rethink this approach in the future. In particular, it's not great that we currently need to restart Scylla to make these metrics available. Scylla already maintains these per-table metrics in per-table memory variables, and we just need a way to optionally expose them through the HTTP request.

To tell the metrics of the different tables apart, each metric's identifier contains the "ks" (keyspace) and "cf" (column family - the old name for table) as labels. For example,

scylla_column_family_pending_compaction{cf="IndexInfo",ks="system",shard="0",type="gauge"} 0.000000

Here we can see the "scylla_column_family_pending_compactions" metric measured in shard 0 of this node, for the table "IndexInfo" in keyspace "system".

[amnonh]

@slivne As I wrote in my answer it is disabled by default and not part of the monitoring stack.

[nyh]

@amnonh this is a feature request... I guess the feature is to enable it? This will require a change to Scylla to allow enabling monitoring per table - at least for a specific table, so this repository might not be the best place for it (do we have an open issue in scylla.git?), but will also require the monitoring stack recognizing and showing these per-table statistics.

[amnonh]

It will probably require support from the metrics layer (seastar) I had a few suggestions for supporting either dynamic enable and disable of metrics or for supporting multiple prometheus endpoint so we can have something like verbosity levels, the default prometheus will return everything, metrics like per-table will be on a different endpoint (different path) and will use the option to select specific metrics in a request.

So far all those attempts got rejected, I've marked this request for future monitoring release.

See also #285