Feat/Add redis twemproxy dashboard

[slopezz]

Related to #196

• Adds redisshard controller suite testing (was one of the few controllers without testing)

• Simplifies grafana dashboard resource by getting all defined keys directly (not necessary struct data)

• Adds a whole new twemproxy grafana dashboard

  • Inspired by our current legacy grafana dashboard deployed on the external prometheus taking metrics from EC2
  • Inspired by other people grafana dashboard https://gist.github.com/VMuliadi/4797e41a5e28300205c1b17604d5ee71
  • Checking official twemproxy exporter metrics, to determine if some metrics are counter or gauges to add the proper promQL query
  • On our legacy dashboard, there were a few metrics treated as counter (with rate) but were gauges, so they have been fixed
  • In addition, after checking official exporter metrics, there have been added a few more interesting graphs that might help on troubleshooting issues

• TwemproxyConfig controller is the only one permitting N CustomResources on the same namespace (all the other controllers harcode resources names), where on TwemproxyConfig CR it is used directly the InstanceName to create the expected target ConfigMap

  • For this reason, for every CR it is created a specific GrafanaDashboard resource by using InstanceName+-twemproxy (to not have collision between different CRs fighting for the resource name), although the dashboard content is exactly the same, and the same information can be extracted for any deployment with twemproxy metrics in any of the created dashboards
    
  • And for the dashboard title inside grafana, it is used the same InstanceName+-twemproxy (grafana-operator uses the internal title in the json file, so need to be also unique for every GrafanaDashboard CR)
    

• TwemproxyConfig controller cannot be checked with ginkgo framework, because it needs more than a k8s API, it needs to connects to sentinel instances (which are connected to redis instances). For this reason, it has been added a simple e2e testing with kuttl tool:

  • Where you define which resources need to be created inside a k8s kind cluster
  • And then check if all needed related k8s resources with specific names has been created and are on ready state (deployment/statefulsets with all ready pods for example).
  • For the TwemproxyConfig, it is needed to create a RedisShard first, then a Sentinel, and finally the TwemproxyConfig
  • Eventually, the twemproxyConfig configmap will be created by TwemproxyConfig controller, and also the related grafana dashboard (by twemproxyConfig controller design,and due to the importance of twemproxy fast reconciliation, if connection to sentinel fails, the twemproxyConfig reconciler never gets to the grafanaDashboard)
  • In the kind-deploy target, the external APIs need to be created before executing the kustomize (and vaildate they are available at API level), because for the specific case of prometheus ServiceMonitor, the kind-deploy target was failing because once created the ServiceMonitor CRD, it was still not available at API level (Wait for a CRD type to deploy before deploying resources that use the type kubernetes/kubectl#1117) , but kustomize was trying to create the saas-operator ServiceMonitor failing with:

  error: unable to recognize "STDIN": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
  

  • This test e2e has been added as a GitHub action for every PR to main, example on this specific PR https://github.com/3scale-ops/saas-operator/runs/6454876406?check_suite_focus=true
    

/home/runner/work/saas-operator/saas-operator/bin/kuttl-v0.12.1 test
=== RUN   kuttl
    harness.go:460: starting setup
    harness.go:251: running tests using configured kubeconfig.
    harness.go:288: Successful connection to cluster at: https://127.0.0.1:35947/
    harness.go:356: running tests
    harness.go:74: going to run test suite with timeout of 240 seconds for each step
    harness.go:368: testsuite: ./test/e2e/ has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/twemproxyconfig
=== PAUSE kuttl/harness/twemproxyconfig
=== CONT  kuttl/harness/twemproxyconfig
    logger.go:42: 15:17:37 | twemproxyconfig | Creating namespace: kuttl-test-charmed-shrimp
    logger.go:42: 15:17:37 | twemproxyconfig/0-install | starting test step 0-install
    logger.go:42: 15:17:38 | twemproxyconfig/0-install | RedisShard:default/shard01 created
    logger.go:42: 15:17:38 | twemproxyconfig/0-install | Sentinel:default/sentinel created
    logger.go:42: 15:17:38 | twemproxyconfig/0-install | TwemproxyConfig:default/backend-twemproxyconfig created
    logger.go:42: 15:19:02 | twemproxyconfig/0-install | test step completed 0-install
    logger.go:42: 15:19:02 | twemproxyconfig | twemproxyconfig events from ns kuttl-test-charmed-shrimp:
    logger.go:42: 15:19:02 | twemproxyconfig | Deleting namespace: kuttl-test-charmed-shrimp
=== CONT  kuttl
    harness.go:402: run tests finished
    harness.go:511: cleaning up
    harness.go:568: removing temp folder: ""
--- PASS: kuttl (86.61s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/twemproxyconfig (84.48s)
PASS

• Redis twemproxy dashboard can be checked at staging
  • Metrics are divided by deployment with a selector
  • First it includes all the twemproxy panels
    
  • And on a seconf part, the related deployment pod metrics (which includes twemproxy container plus other pod containers like app or envoy)
    
  • If by any reason toy want to see specific pod twemproxy contaienr metrics, you can use the associated drill down link to the Pod Details dashboard
    

/kind feature
/priority important-soon
/assign

[raelga]

Awesome job.
/lgtm

[3scale-robot]

LGTM label has been added.

Git tree hash: bd65d5ca888db6ec51d73cb2d8de411a6f859355

[slopezz]

/approve

[3scale-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [slopezz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment