Skip to content

Commit

Permalink
change metrics example to use servicemonitor
Browse files Browse the repository at this point in the history
  • Loading branch information
@jcmoraisjr
jcmoraisjr committed May 7, 2022
1 parent 346c94f commit 18d2686
Show file tree
Hide file tree
Showing 6 changed files with 2,373 additions and 2,677 deletions.
261 changes: 115 additions & 146 deletions docs/content/en/docs/examples/metrics.md
View file
Expand Up @@ -6,198 +6,165 @@ description: >
Demonstrate how to collect and expose ingress controller and haproxy metrics.
---

{{% pageinfo %}}
This is a `v0.10` example and need HAProxy Ingress `v0.10-snapshot.5` or above
{{% /pageinfo %}}

This example demonstrates how to configure [Prometheus](https://prometheus.io) to collect ingress controller and haproxy metrics, and also to configure a [Grafana](https://grafana.com) dashboard to expose these metrics.
This example demonstrates how to configure [Prometheus](https://prometheus.io) and [Grafana](https://grafana.com) to collect and expose HAProxy and HAProxy Ingress metrics using [Prometheus Operator](https://prometheus-operator.dev).

## Prerequisites

This document has the following prerequisite:
This document requires only a Kubernetes cluster. HAProxy Ingress doesn't need to be installed, and if so, the installation process should use the [Helm chart]({{% relref "/docs/getting-started#installation" %}}).

* A Kubernetes cluster with a running HAProxy Ingress controller v0.10 or above. See the [getting started]({{% relref "../getting-started" %}}) guide.
## Configure Prometheus Operator

## Configure the controller
This section can be skipped if the Kubernetes cluster has already a running Prometheus Operator.

HAProxy Ingress by default does not configure the haproxy's prometheus exporter. The patch below configures the haproxy's internal prometheus exporter in the port `9105`:
HAProxy Ingress installation configures Prometheus using a ServiceMonitor custom resource. This resource is used by [Prometheus Operator](https://prometheus-operator.dev) to configure Prometheus instances. The following steps deploy Prometheus Operator via [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart.

```
kubectl --namespace ingress-controller patch configmap haproxy-ingress -p '{"data":{"prometheus-port":"9105"}}'
```
Create a file named `prometheus-operator-values.yaml` - change both hostnames with a name that resolves to the Kubernetes cluster:

The following patch adds ports `9105` and `10254` to the HAProxy Ingress container. The port declaration is used by the Prometheus' service discovery:
```yaml
grafana:
enabled: true
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: haproxy
hosts:
- grafana.192.168.0.11.nip.io
tls:
- hosts:
- grafana.192.168.0.11.nip.io
```

Note: this patch will restart the controller!
Add `kube-prometheus-stack` helm repo:

```
kubectl --namespace ingress-controller patch deployment haproxy-ingress -p '{"spec":{"template":{"spec":{"containers":[{"name":"haproxy-ingress","ports":[{"name":"exporter","containerPort":9105},{"name":"ingress-stats","containerPort":10254}]}]}}}}'
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
```

## Deploy Prometheus

This will create a Prometheus deployment with no resource limits, a configuration file which will scrape haproxy and also HAProxy Ingress metrics every `10s`, and also a role and rolebinding which allows Prometheus to discover haproxy and controller endpoints using k8s:
Install the chart:

```
kubectl create -f https://haproxy-ingress.github.io/docs/examples/metrics/prometheus.yaml
helm install prometheus prometheus-community/kube-prometheus-stack\
--create-namespace --namespace monitoring\
-f prometheus-operator-values.yaml
```

{{% alert title="Note" %}}
This deployment has no persistent volume, so all the collected metrics will be lost if the pod is recreated.
Bitnami has also a Prometheus Operator [helm chart](https://github.com/bitnami/charts/tree/master/bitnami/kube-prometheus) and it's also a good option. Note however that the values file has a different syntax.
{{% /alert %}}

{{% alert title="Warning" color="warning" %}}
If HAProxy Ingress wasn't deployed with Helm, change the following line in the `configmap/prometheus-cfg` resource, jobs `haproxy-ingress` and `haproxy-exporter`:
## Configure HAProxy Ingress

```diff
relabel_configs:
- - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
+ - source_labels: [__meta_kubernetes_pod_label_run]
regex: haproxy-ingress
```

This will ensure that Prometheus finds the controller pods.
{{% /alert %}}
The steps below configures HAProxy Ingress' Helm chart to add a new ServiceMonitor custom resource. This resource will be responsible for HAProxy and HAProxy Ingress metrics scrape.

Check if Prometheus is up and running:

```
kubectl --namespace ingress-controller get pod -lrun=prometheus -w
```

Check also if Prometheus found the haproxy and the controller endpoints:

```
kubectl --namespace ingress-controller port-forward svc/prometheus 9090:9090
Merge the content below to the actual `haproxy-ingress-values.yaml` file:
```yaml
controller:
stats:
enabled: true
metrics:
enabled: true
serviceMonitor:
enabled: true
labels:
release: prometheus
metrics:
relabelings:
- replacement: cl1
targetLabel: cluster
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: hostname
ctrlMetrics:
relabelings:
- replacement: cl1
targetLabel: cluster
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: hostname
```

There are two important configurations in the snippet above:

* Added a label `release: prometheus` in the ServiceMonitor. HAProxy Ingress metrics will share the same Prometheus instance installed by Prometheus Operator. This can be changed to another dedicated instance, and must be checked if using another customized Prometheus Operator deployment.
* Added relabels to HAProxy and HAProxy Ingress metrics. The HAProxy Ingress dashboard uses `hostname` label as a way to distinguish two controller instances, and also `cluster` label to distinguish controllers running on distinct clusters. The source of the name can be adjusted but the label name should be the same.

Now upgrade the chart - change `upgrade` to `install` if HAProxy Ingress isn't installed yet:
```
helm upgrade haproxy-ingress haproxy-ingress/haproxy-ingress\
--create-namespace --namespace ingress-controller\
-f haproxy-ingress-values.yaml
```

Open [localhost:9090/targets](http://127.0.0.1:9090/targets) in your browser, all haproxy and controller instances should be listed, up, and green.

## Deploy Grafana

The following instruction will create a Grafana deployment with no resource limit, and also its service:
## Compatibility

```
kubectl create -f https://haproxy-ingress.github.io/docs/examples/metrics/grafana.yaml
```
This dashboard works with HAProxy's internal Prometheus exporter. Follow these steps to adjust the scrape config and the dashboard if using [Prometheus' HAProxy Exporter](https://github.com/prometheus/haproxy_exporter):

Check if Grafana is up and running:
Change the metric name of "Backend status / Top 5 max/avg connection time" to `haproxy_backend_http_connect_time_average_seconds`

```
kubectl --namespace ingress-controller get pod -lrun=grafana -w
```

Create the ingress which will expose Grafana. Change `HOST` below to a domain of the cluster, or just change the inner IP number to the IP of the HAProxy Ingress node:

```
HOST=grafana.192.168.1.1.nip.io
kubectl create -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: ingress-controller
spec:
rules:
- host: $HOST
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
tls:
- hosts:
- $HOST
EOF
Add this relabel configuration in the `haproxy-ingress-values.yaml` file
```yaml
controller:
...
serviceMonitor:
...
metrics:
...
metricRelabelings:
- sourceLabels: [frontend]
targetLabel: proxy
- sourceLabels: [backend]
targetLabel: proxy
```

## Configure dashboard
## Configure the dashboard

Now its time to see what Prometheus is collecting. Grafana can be acessed in the `HOST` used to configure the ingress object.
Import [this](https://grafana.com/grafana/dashboards/12056) Grafana dashboard. If Grafana was deployed using the steps provided in this walktrought:

* Log in to Grafana, user is `admin` and the first password is `admin`
* Create a Prometheus data source, the endpoint is `http://prometheus:9090`
* Click the big `+` plus sign in the left side, and Import [this](https://grafana.com/grafana/dashboards/12056) dashboard
* Open Grafana page - the URL is the same provided in the `prometheus-operator-values.yaml` file and should resolve to the ingress deployment
* Log in to Grafana, user is `admin` and the first password is `prom-operator`
* Click the big plus `+` sign in the left side, Import, type `12056` as the Grafana.com ID, Load, select a Prometheus datasource, Import

{{% alert title="Note" %}}
This deployment has no persistent volume, so all customizations will be lost if the pod is recreated.
{{% /alert %}}

If everything worked as expected, you should see a dashboard like this:
If everything worked as expected, the dashboard should look like this:

![](/docs/examples/metrics/dashboard-1.png)

## Compatibility

Check if you have any of these scenarios and adjust Prometheus or Grafana accordingly.

**Grafana < 6.5**

Use Grafana 6.5 or higher. Although an older Grafana won't refuse to install this dashboard, some widgets will not render as expected on older versions.

**The `hostname` label**

All the metrics of this dashboard are grouped by the `hostname` label. If you already have Prometheus and HAProxy Exporter, ensure that the `hostname` label uniquely identifies all HAProxy Ingress instances, just like the proposed Prometheus configuration does.

**The `_front__tls` proxy**

This dashboard is designed to work without the `_front__tls` proxy, such proxy is created whenever ssl-passthrough is used or timeout client is configured in the ingress object. Just to be sure, check if `grep _front__tls /etc/haproxy/haproxy.cfg` find a proxy declaration. If the `_front__tls` proxy exists, edit the dashboard and change the variable `$public_frontend` to the following new value: `(_tcp_.*|_front_http|_front__tls)`

**Using the Prometheus' HAProxy Exporter**

This dashboard works with haproxy's internal Prometheus exporter. Follow these steps to adjust the scrape config and the dashboard if using [Prometheus' HAProxy Exporter](https://github.com/prometheus/haproxy_exporter):

* Change the metric name of "Backend status / Top 5 max/avg connection time" to `haproxy_backend_http_connect_time_average_seconds`
* Add this `metric_relabel_config` in the Prometheus configuration:

```yaml
metric_relabel_configs:
- source_labels: [frontend]
regex: (.+)
target_label: proxy
- source_labels: [backend]
regex: (.+)
target_label: proxy
```

## Test

Lets make some noise and see what the dashboard tell us about our HAProxy Ingress cluster.

Deploy a demo application and a custom (self-signed) certificate:

```
openssl req -x509 -subj "/CN=whoami.localdomain" -nodes -days 30 -newkey rsa:2048 -keyout /tmp/w.key -out /tmp/w.crt
kubectl --namespace default create secret tls whoami --cert /tmp/w.crt --key /tmp/w.key
rm -fv /tmp/w.crt /tmp/w.key
kubectl --namespace default create deploy whoami --image jcmoraisjr/whoami
kubectl --namespace default scale deploy whoami --replicas=4
kubectl --namespace default expose deploy whoami --port 8000
kubectl create -f https://haproxy-ingress.github.io/docs/examples/metrics/whoami-ingress.yaml
openssl req -x509 -subj "/CN=dory.localdomain" -nodes -days 30 -newkey rsa:2048 -keyout /tmp/h.key -out /tmp/h.crt
kubectl --namespace default create secret tls dory --cert /tmp/h.crt --key /tmp/h.key
rm -fv /tmp/h.crt /tmp/h.key
kubectl --namespace default create deploy dory --image jcmoraisjr/dory
kubectl --namespace default scale deploy dory --replicas=4
kubectl --namespace default expose deploy dory --port 8000
kubectl --namespace default create ingress dory\
--annotation kubernetes.io/ingress.class=haproxy\
--annotation haproxy-ingress.github.io/ssl-redirect=false\
--rule="dory.localdomain/*=dory:8000,tls=dory"
```

Check if the app is up and running:

```
kubectl --namespace default get pod -lapp=whoami -w
kubectl --namespace default get pod -lapp=dory -w
```

Download [vegeta](https://github.com/tsenart/vegeta/releases) and place it in the path.

Make a test and check if everything is working as expected. Change IP below to the IP of a HAProxy Ingress node:

```
IP=192.168.1.1
echo "GET http://${IP}" |\
vegeta attack -duration=1s -rate=1 -header "Host: whoami.localdomain" -keepalive=true |\
IP=192.168.0.11
## Using Fish?
# set IP 192.168.0.11
echo "GET http://$IP" |\
vegeta attack -duration=1s -rate=1 -header "Host: dory.localdomain" -keepalive=true |\
vegeta report
```

You should see something like this. The most important part is Success ratio=100% and an Error Set empty:
The output should look like this. The most important part is Success ratio=100% and an Error Set empty:

```
...
Expand All @@ -209,28 +176,30 @@ Error Set:
Now the real test. Adjust the duration and rate (number of requests per second) if needed. A dual core VM dedicated to HAProxy Ingress should accept a few thousands requests per second. Lets configure `200` which should move some lines in the dashoard:

```
IP=192.168.1.1
echo "GET http://${IP}" |\
vegeta attack -duration=5m -rate=200 -header "Host: whoami.localdomain" -keepalive=true |\
IP=192.168.0.11
echo "GET http://$IP" |\
vegeta attack -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=true |\
vegeta report
```

Follow the dashboard while the test is running. Most metrics have its resolution in `1m` (one minute) so you should wait this time to see the correct conn/s, rps, proc use and so on.
Follow the dashboard while the test is running. Most metrics have its resolution in `1m` (one minute) so wait at least this amount of time to see the correct conns/s, rps, proc use and so on.

What's the impact of not use keepalive? Try the same test, changing only `-keepalive` to `false`:
What's the impact of not using keepalive? Try the same test, changing only `-keepalive` to `false`:

```
IP=192.168.1.1
echo "GET http://${IP}" |\
vegeta attack -duration=5m -rate=200 -header "Host: whoami.localdomain" -keepalive=false |\
IP=192.168.0.11
ulimit -n 2048 # avoids 'too many open files' error in the client side
echo "GET http://$IP" |\
vegeta attack -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=false |\
vegeta report
```

Last test: what about TLS connections without keepalive? Change `http` to `https` and add `-insecure` command-line option to Vegeta:

```
IP=192.168.1.1
echo "GET https://${IP}" |\
vegeta attack -insecure -duration=5m -rate=200 -header "Host: whoami.localdomain" -keepalive=false |\
IP=192.168.0.11
ulimit -n 2048
echo "GET https://$IP" |\
vegeta attack -insecure -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=false |\
vegeta report
```
Binary file modified docs/content/en/docs/examples/metrics/dashboard-1.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 0 additions & 32 deletions docs/content/en/docs/examples/metrics/grafana.yaml
View file

This file was deleted.

0 comments on commit 18d2686

Please sign in to comment.