[cassandra] Add support for Prometheus/Grafana monitoring

[yanniszark]

Is this a bug report or feature request?

• Feature Request

What should the feature do:
The cassandra operator should install the necessary plugins, in order for cassandra to export prometheus metrics. We only use plugins that will work as a javaagent. Currently there are various plugins doing that:

• cassandra-exporter by Instaclustr: Seems to be the best solution but is still a very early project with no Grafana dashboard. If it matures and is maintained, this will most likely be the right choice.
• jmx-exporter by Prometheus: Not made specifically for Cassandra, but it does the job. There is also a Grafana dashboard for it.

What is use case behind this feature:

Users should be able to monitor their Cassandra cluster.
Unfortunately, there's no clear cut solution.

⚠️ If you have experience monitoring Cassandra using Prometheus and Grafana, please tell us your opinion.

[umangachapagain]

The cassandra operator should install the necessary plugins, in order for cassandra to export prometheus metrics. We only use plugins that will work as a javaagent. Currently there are various plugins doing that:

* [cassandra-exporter by Instaclustr](https://github.com/instaclustr/cassandra-exporter): Seems to be the best solution but is still a very early project with no Grafana dashboard. If it matures and is maintained, this will most likely be the right choice.

Not experienced with Cassandra monitoring or cassandra-exporter, but with little experience that I have on monitoring with Prometheus & Grafana, cassandra-exporter seems like a safe bet on long term.
Alerting and Dashboards can be maintained as a separate project and can mature parallel to the exporter.

This is the way kubernetes-mixin does it, and we at gluster-mixins do it.

[yanniszark]

@umangachapagain thanks for the input!
cassandra-exporter seems promising indeed, we could try to integrate it.

An outline of the changes for this feature is:

1. Add cassandra-exporter plugin in the operator image (images/cassandra/Dockerfile).
2. Add cassandra-exporter as a javaagent (-javaagent:<plugin-dir-name>/cassandra-exporter.jar=--listen=:9180) in the sidecar's config setup code.
3. Add prometheus annotations:
   • prometheus.io/scrape
   • prometheus.io/port

to the StatefulSet's PodSpec for port 9180.

After that we can try it out with a basic helm installation of Prometheus.

[umangachapagain]

@yanniszark I could look into it. But, since I do not have any idea on cassandra-exporter and javaagent etc, I will require some help setting things up and it definitely will take some time to have a complete feature.

[nabokihms]

@BlaineEXE is work on the issue in progress? I can take a look and do necessary changes if nobody minds.

[yanniszark]

I should give a little bit more context around this issue, since I have tried some things.
The problem isn't so much in integrating an exporter as in finding a good Grafana dashboard to go along with it.

The problem with the Cassandra metrics is that when exporting to Prometheus, the names aren't standard.
So two different exporters might generate a slightly different name for the same metric.
This breaks compatibility for Grafana dashboards, since they depend on the metric names.
In other words, a dashboard made for jmx-exporter might not work for cassandra-exporter.

I tried to integrate cassandra-exporter a while back, but didn't find any compatible dashboard.
Considering that:

• cassandra-exporter hasn't seen much activity since its launch and also isn't compatible with many Cassandra versions
• cassandra-exporter has no compatible Grafana dashboard

it seems logical to reject it as an option for now and go for jmx-exporter.
It has a compatible Grafana dashboard, can be run as a javaagent and is a mature project.

@nabokihms it would be great if you could give us a hand in getting this feature into Cassandra operator! 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nabokihms]

I think we need to keep the issue opened because of the pull request above.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.