Metrics that would be good to have

[elsmorian]

After a chat with @adejanovski about if Reaper exported metrics, he mentioned its currently in master but not many metrics were exported yet, and if I had any requests to make a GitHub issue. For us, useful metrics would be:

• Number of segments pending to be repaired (so we can chart progress over time, similar to Cassandra's own Repair PendingTasks metric
• Number of segments repaired per second (likely to be low for big repairs but still handy)
• Number of postponed repair events due to high load / repairs already running per second.

[elsmorian]

Oh, and the current number of nodes in each data centre that are up or down, that would be super helpful!

[rzvoncek]

Hi!

I'll look into this one. And while I'm at it, I'll add two more metrics:

• repair progress (per cluster): once plotted in a dashboard, repair progress being flat will nicely show stalled repairs.
  • This is somewhat similar to the number of segments above, let me see which one (or both) to include.
• time of last successful repair (per cluster): will make it easy to spot missing repairs on cluster

[elsmorian]

@rzvoncek 👍 totally agree on that, thanks for having a look into this!

[rzvoncek]

Hi.

I've ended up not adding the postpones metric, because there already is something similar:

"io.cassandrareaper.service.SegmentRunner.postpone.null.testcluster.keyspace1" : {
      "count" : 13
    },

The null should be a coordinator host, but for some reason it doesn't populate for me :-/.

[elsmorian]

Thanks for adding these in :)