docs: monitoring setup guide

docs/source/generic.md

@@ -275,46 +275,9 @@ kubectl create secret -n scylla generic scylla-client-config-secret --from-file
 ```
 After a restart the operator will use the security token when it interacts with scylla via the agent.
 
- ### Setting up Monitoring
+ ### Set up monitoring
 
- Both Prometheus and Grafana were configured with specific rules for Scylla Operator.
- Both of them will be available under the `monitoring` namespace. 
- Customization can be done in `examples/common/prometheus/values.yaml` and `examples/common/grafana/values.yaml`.
-
-1. Create the monitoring namespace
-    ```console
-    kubectl create namespace monitoring
-    ```
-1. Add Prometheus charts repository
-   ```console
-   helm repo add stable  https://charts.helm.sh/stable
-   helm repo update
-   ```
-1. Install Prometheus
-    ```console
-    helm upgrade --install scylla-prom --namespace monitoring stable/prometheus -f examples/common/prometheus/values.yaml
-    ```
-    If you want to tweak the prometheus properties, for example it's assigned memory, you can override it by adding a command line argument like this: `--set server.resources.limits.memory=4Gi`
-
-1. Install Grafana
-    First you need to prepare the dashboards to make them available in Grafana. 
-    You can do this by running the following command in the `examples` directory:
-    ```console
-    ./dashboards.sh
-    ```
-    __NB__: Keep in mind that this is a test setup. For production use, check grafana and prometheus helm chart page for advanced deployment instructions.
-
-    Now the dashboards can be created along with the grafana plugin like this:
-    ```console
-    helm upgrade --install scylla-graf --namespace monitoring stable/grafana -f examples/common/grafana/values.yaml
-    ```
- 
-    To access Grafana locally, run:
-    ```
-    kubectl --namespace monitoring port-forward $(kubectl get pods -n monitoring -l "app.kubernetes.io/instance=scylla-graf" -o jsonpath="{.items[0].metadata.name}") 3000
-    ```
- 
-    You can find it on `http://0.0.0.0:3000` and login with the credentials `admin`:`admin`.
+To set up monitoring using Prometheus and Grafana follow [this guide](monitoring.md).
 
 ## Scale Up
 

docs/source/index.rst

@@ -10,6 +10,7 @@ Scylla Operator Documentation
    eks
    gke
    manager
+   monitoring
    node_operations
    upgrade
    scylla_cluster_crd
@@ -45,6 +46,7 @@ Currently it supports:
 * :doc:`Deploying Scylla on EKS (experimental) <eks>`
 * :doc:`Deploying Scylla on GKE <gke>`
 * :doc:`Deploying Scylla Manager on a Kubernetes Cluster <manager>`
+* :doc:`Setting up Monitoring using Prometheus and Grafana <monitoring>`
 * :doc:`Node operations <node_operations>`
 * :doc:`Upgrade procedures <upgrade>`
 * :doc:`Scylla Cluster Custom Resource Definition (CRD) <scylla_cluster_crd>`

docs/source/monitoring.md

@@ -0,0 +1,65 @@
+### Setting up Monitoring
+
+Both Prometheus, Grafana and AlertManager were configured with specific rules for Scylla Monitoring.
+All of them will be available under the `scylla-monitoring` namespace.
+Customization can be done in `examples/common/monitoring/values.yaml`
+
+1. Add monitoring stack charts repository
+   ```
+   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+   helm repo update
+   ```
+1. Install monitoring stack
+    ```
+    helm install monitoring prometheus-community/kube-prometheus-stack --values examples/common/monitoring/values.yaml --create-namespace --namespace scylla-monitoring
+    ```
+   If you want to tweak the prometheus properties, for example it's assigned memory,
+   you can override it by adding a command line argument like this: `--set prometheus.resources.limits.memory=4Gi`
+   or edit values file located at `examples/common/monitoring/values.yaml`.
+
+1. Install Service Monitors
+
+    ServiceMonitors are used by the Prometheus to discover applications exposing metrics.
+
+    ```
+    # Scylla Service Monitor
+    kubectl apply -f examples/common/monitoring/scylla-service-monitor.yaml
+
+    # Scylla Manager Service Monitor
+    kubectl apply -f examples/common/monitoring/scylla-manager-service-monitor.yaml
+    ```
+
+1. Download dashboards
+
+   First you need to download the dashboards to make them available in Grafana.
+   You can do this by running the following command:
+    ```
+    wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-3.6.0.tar.gz
+    tar -xvf scylla-monitoring-3.6.0.tar.gz
+    ```
+
+1. Install dashboards
+
+    Scylla Monitoring comes with pre generated dashboards suitable for multiple Scylla versions.
+    In this example we will use dashboards for Scylla 4.3, and Scylla Manager 2.2.
+    Amend directory path to generated dashboards to version suitable for your deployment.
+
+   Now the dashboards can be created like this:
+    ```
+    # Scylla dashboards
+    kubectl -n scylla-monitoring create configmap scylla-dashboards --from-file=scylla-monitoring-scylla-monitoring-3.6.0/grafana/build/ver_4.3
+    kubectl -n scylla-monitoring patch configmap scylla-dashboards  -p '{"metadata":{"labels":{"grafana_dashboard": "1"}}}'
+
+    # Scylla Manager dashboards
+    kubectl -n scylla-monitoring create configmap scylla-manager-dashboards --from-file=scylla-monitoring-scylla-monitoring-3.6.0/grafana/build/manager_2.2
+    kubectl -n scylla-monitoring patch configmap scylla-manager-dashboards  -p '{"metadata":{"labels":{"grafana_dashboard": "1"}}}'
+    ```
+
+    Once Grafana sidecar picks up these dashboards they should be accessible in Grafana.
+
+To access Grafana locally, run:
+    ```
+    kubectl -n scylla-monitoring port-forward deployment.apps/monitoring-grafana 3000
+    ```
+
+   You can find it on `http://127.0.0.1:3000` and login with the credentials `admin`:`admin`.

examples/common/monitoring/scylla-manager-service-monitor.yaml

@@ -14,4 +14,4 @@ spec:
         - sourceLabels: [ host ]
           targetLabel: instance
           regex: (.*)
-          replacement: ${1}
+          replacement: ${1}

examples/common/monitoring/scylla-service-monitor.yaml

@@ -30,4 +30,4 @@ spec:
           targetLabel: dc
           regex: (.*)
           replacement: ${1}
-          action: replace
+          action: replace