cassandra: JMX prometheus exporter

[nabokihms]

Export prometheus metrics for cassandra in sidecar container

Signed-off-by: Maksim Nabokikh maksim.nabokikh@flant.com

Description of your changes:
Add jmx exporter with standard configuration to sidecar container.

Test steps:

• Start minikube with tests/scripts/minikube.sh up
• Deploy cassandra from cluster/examples/kubernetes/cassandra
• Get the metrics with curl

This dashboard seems ok https://grafana.com/dashboards/5408 for monitoring

Which issue is resolved by this Pull Request:
Resolves #2530

Checklist:

• Documentation has been updated, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.
• Comments have been added or updated based on the standards set in CONTRIBUTING.md

[test cassandra]

[galexrt]

Would be good if you could verify the Prometheus annotations position with, e.g., prometheus-operator to see that Prometheus picks the Pods up.

[nabokihms]

There is new option in cluster.cassandra CRD, which provides the ability to specify ConfigMap for jmx-prometheus-exporter for Cassandra Rack.

[nabokihms]

@travisn @galexrt I will appreciate any feedback for this PR.

[yanniszark]

@nabokihms this is excellent!
Thanks a lot for your effort, I just found out about this PR.
I will put some cycles of my spare time in reviewing this PR so we can get it merged asap 😄

[yanniszark]

@nabokihms thanks a lot for your work!
I have a few comments on the sidecar code, but other than that it looks great.

In addition, ConfigMap updates are currently not handled. This means that if a ConfigMap is updated, changes will be propagated but the Pods won't be restarted (which I assume is necessary for jmx_exporter to pick up the changes).

We can either:

1. Implement an update mechanism. For example, save the ConfigMap SHA-256 hash in a label of each StatefulSet's PodTemplate. This way, a change in the ConfigMap hash will trigger a rolling update across all Racks.
2. Leave that for later and instead document how to change the ConfigMap manually. The process is
   • Apply new ConfigMap.
   • Rolling restart all Racks:

     statefulsets=[...]
     for name in statefulsets:
         kubectl rollout restart statefulset $name -n $namespace

[yanniszark]

@nabokihms any update on this?
Do you plan to continue the work on this PR? Would be awesome to get this in! 😄

[nabokihms]

@yanniszark I will finish this till weekends.

[travisn]

@nabokihms How is this looking? We are starting to lock down for the 1.2 release by the end of the week. Thanks!

[nabokihms]

Thanks for informing me. We want to merge it in 1.2. I will try my best to end work on the PR till tomorrow.

[nabokihms]

I made the necessary changes and also updated the docs to make difficult cases clear.

[paulcostinean]

Is this getting any more love soon? Would be nice to have the option to use the jmx exporter and it looks like 90% of the work is already done 🚀

[yanniszark]

@paulcostinean thanks for the ping! :)
I'll try to take a look this weekend.

[yanniszark]

@nabokihms thanks a lot for your effort!
Overall, this looks ready to merge, only very small things remaining:

• Rebase on latest master and resolve merge conflict of Release Notes.

• Since this PR introduces a new field, you have to regenerate the Cassandra client code (Deepcopy methods). The procedure I followed is:

rm -rf vendor
GO111MODULE=off make vendor
GO111MODULE=off make codegen

• The test.go file has a newline has changed (newline) while it shouldn't be touched by this PR. Can you revert that change?

• Add some commands to help the user restart their Cassandra Cluster in the event of a ConfigMap change:

NAMESPACE=<namespace>
CLUSTER=<cluster_name>

RACKS=$(kubectl get sts -n ${NAMESPACE} -l "cassandra.rook.io/cluster=${CLUSTER}")
echo ${RACKS} | xargs -n1 kubectl rollout restart -n ${NAMESPACE}

To make it more comfortable, I included these changes in a review branch, so you can see what I mean:
https://github.com/yanniszark/rook/tree/cassandra-jmx-metrics-review

After that, we should be ready to merge!

[ghost]

There were the following issues with this Pull Request

• Commit: 7d9a62b
  • ✖ message may not be empty
  • ✖ type may not be empty

You may need to change the commit messages to comply with the repository contributing guidelines.

---

🤖 This comment was generated by commitlint[bot]. Please report issues here.

Happy coding!

[nabokihms]

@yanniszark Thanks a lot for the PR review and especially for the review branch! It helps a lot. I merged all the changes to the current branch.

[yanniszark]

@nabokihms thanks for all the great work!
I tested and everything looks good.
You just have a small bug that was caught by a unit test (that's why the CI was failing).

More specifically, you need to check if the jmxExporterConfigMapName field is nil before dereferencing.
I have included comments in the relevant parts.

Other than that, we're good to go!
/lgtm
/approve

If needed, I will approve again after the necessary changes.
Let's get this through the door!

[yanniszark]

I see errors in two tests: TestCephHelmSuite and TestCephBlockSuite.
I suspect those are flakes and shouldn't be failing as the code changes in this PR are irrelevant to them.
@nabokihms can you make sure this is rebased on latest master?

cc @travisn @galexrt