Skip to content

Commit

Permalink
docs: Document how to set up monitoring and logging on OpenShift when…
Browse files Browse the repository at this point in the history 
… using the operator (#959)

* Add docs for monitoring on OpenShift using operator

* Add instructions for setting pod annotations using the operator

This is useful to add Prometheus annotations

* Add instructions for enabling logging using the operator
  • Loading branch information
@rm3l
rm3l committed Feb 16, 2024
1 parent d0c68b1 commit 0119b18
Showing 1 changed file with 129 additions and 10 deletions.
139 changes: 129 additions & 10 deletions showcase-docs/monitoring-and-logging.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,30 @@

The Backstage Showcase provides a `/metrics` endpoint that provides Prometheus metrics about your backstage application. This endpoint can be used to monitor your backstage instance using Prometheus and Grafana.

When deploying Backstage Showcase onto a kubernetes cluster with the [RHDH Helm chart](https://github.com/redhat-developer/rhdh-chart), monitoring and logging for your RHDH instance can be configured using the following steps.
When deploying Backstage Showcase onto a kubernetes cluster with the [RHDH Helm chart](https://github.com/redhat-developer/rhdh-chart) or the [RHDH Operator](https://github.com/janus-idp/operator), monitoring and logging for your RHDH instance can be configured using the following steps.

## Prerequisites

- Kubernetes 1.19+
- Helm 3.2.0+
- PV provisioner support in the underlying infrastructure
- The [RHDH Helm chart repositories](https://github.com/redhat-developer/rhdh-chart#installing-from-the-chart-repository) have been added
- If using the Helm Chart
- Helm 3.2.0+
- The [RHDH Helm chart repositories](https://github.com/redhat-developer/rhdh-chart#installing-from-the-chart-repository)

## Metrics Monitoring

### Enabling Metrics Monitoring on Openshift

To enable metrics on Openshift, you will need to modify the `values.yaml` of the [RHDH Helm chart](https://github.com/redhat-developer/rhdh-chart/blob/main/charts/backstage/values.yaml)
To enable metrics monitoring on OpenShift, we need to create a `ServiceMonitor` resource in the OpenShift cluster that will be used by Prometheus to scrape metrics from your Backstage instance. For the metrics to be ingested by the built-in Prometheus instances in Openshift, please ensure you enabled [monitoring for user-defined projects](https://docs.openshift.com/container-platform/latest/monitoring/enabling-monitoring-for-user-defined-projects.html).

#### Helm deployment

To enable metrics on Openshift when deploying with the [RHDH Helm chart](https://github.com/redhat-developer/rhdh-chart), you will need to modify the [`values.yaml`](https://github.com/redhat-developer/rhdh-chart/blob/main/charts/backstage/values.yaml) of the Chart.

To obtain the `values.yaml`, you can run the following command:

```bash
helm show values janus-idp/backstage > values.yaml
helm show values redhat-developer/backstage > values.yaml
```

Then, you will need to modify the `values.yaml` to enable metrics monitoring by adding the following configurations:
Expand All @@ -37,14 +42,47 @@ upstream:
Then you can deploy the Janus Helm chart with the modified `values.yaml`:

```bash
helm upgrade -i <release_name> janus-idp/backstage -f values.yaml
helm upgrade -i <release_name> redhat-developer/backstage -f values.yaml
```

This will create a `ServiceMonitor` resource in your Openshift cluster that will be used by Prometheus to scrape metrics from your Backstage instance. For the metrics to be ingested by the built-in Prometheus instances in Openshift, please ensure you enabled [monitoring for user-defined projects](https://docs.openshift.com/container-platform/latest/monitoring/enabling-monitoring-for-user-defined-projects.html)

You can then verify metrics are being captured by navigating to the Openshift Console. Go to `Developer` Mode, change to the namespace the showcase is deployed on, selecting `Observe` and navigating to the `Metrics` tab. Here you can create PromQL queries to query the metrics being captured by Prometheus.
![Openshift Metrics](./images/openshift-metrics.png)

#### Operator-backed deployment

At the moment, the operator does not support creating OpenShift `ServiceMonitor` instances out of the box.

However, you can manually create a `ServiceMonitor` resource for your operator-backed Backstage instance; and it will be used by Prometheus to scrape metrics from your Backstage instance. To do so, please adjust and run the following commands using either `oc` or `kubectl`:

```bash
# make sure to update CR_NAME value accordingly
$ CR_NAME=my-rhdh
$ MY_PROJECT=my-project
$ cat <<EOF > /tmp/${CR_NAME}.ServiceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ${CR_NAME}
namespace: ${MY_PROJECT}
labels:
app.kubernetes.io/instance: ${CR_NAME}
app.kubernetes.io/name: backstage
spec:
namespaceSelector:
matchNames:
- ${MY_PROJECT}
selector:
matchLabels:
janus-idp.io/app: backstage-${CR_NAME}
endpoints:
- port: http-backend
path: '/metrics'
EOF
$ oc apply -f /tmp/${CR_NAME}.ServiceMonitor.yaml
```

Similar to the instructions above for a Helm-based deployment, you can then verify metrics are being captured by navigating to the Openshift Console. Go to `Developer` Mode, change to the namespace the showcase is deployed on, selecting `Observe` and navigating to the `Metrics` tab.

### Enabling Metrics Monitoring on Azure Kubernetes Service (AKS)

To enable metrics monitoring for Backstage Showcase on Azure Kubernetes Service (AKS), you can use the [Azure Monitor managed service for Prometheus](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-overview). The AKS cluster will need to have an associated [Azure Monitor workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-enable?tabs=azure-portal).
Expand All @@ -53,7 +91,11 @@ One method is to configure the metrics scraping of your AKS cluster using the [A

The other method is to configure the Azure Monitor _monitoring_ add-on which also allows you to [send Prometheus metrics to the Log Analytics workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-logs). These metrics can then be queried using [Log Analytics queries](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query#prometheus-metrics) as well as be visible in a Grafana instance.

In both methods, we can configure the metrics scraping to scrap from pods based on pod Annotations. To add annotations to the backstage pod, add the following to the Janus Helm chart `values.yaml`:
In both methods, we can configure the metrics scraping to scrap from pods based on pod Annotations. Follow the steps below depending on how the Backstage Showcase application is deployed.

#### Helm deployment

To add annotations to the backstage pod, add the following to the Janus Helm chart `values.yaml`:

```yaml title="values.yaml"
upstream:
Expand All @@ -67,6 +109,65 @@ upstream:
prometheus.io/scheme: 'http'
```

#### Operator-backed deployment

For an operator-backed deployment, you will need to first create your Custom Resource, then manually add the annotations to the Backstage Pod created by the Operator.

Using `oc` or `kubectl`, you can run the following command:

```bash
# make sure to update CR_NAME value accordingly
$ CR_NAME=my-rhdh
$ oc annotate pods \
--selector janus-idp.io/app="backstage-${CR_NAME}" \
prometheus.io/scrape='true' \
prometheus.io/path='/metrics' \
prometheus.io/port='7007' \
prometheus.io/scheme='http'
```

**NOTE**: Please note that these annotations might be lost if the pod has to be recreated by the operator. In this case, you can either add the annotations again or, as an administrator of the operator, change its default configuration.
To have something persistent, if you have access to the namespace where the operator is running (usually `rhdh-operator`, `openshift-operators`, or `backstage-system`), you can do the following, using either `oc` or `kubectl`:

1. Edit the default configuration of the operator:

```bash
# make sure to update OPERATOR_NS accordingly
$ OPERATOR_NS=rhdh-operator
$ oc edit configmap backstage-default-config -n "${OPERATOR_NS}"
```

2. Find the `deployment.yaml` key in the ConfigMap, and add the annotations to the `spec.template.metadata.annotations` field, like so:

```yaml
deployment.yaml: |-
apiVersion: apps/v1
kind: Deployment
# --- truncated ---
spec:
template:
# --- truncated ---
metadata:
labels:
janus-idp.io/app: # placeholder for 'backstage-<cr-name>'
# --- truncated ---
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
prometheus.io/port: '7007'
prometheus.io/scheme: 'http'
# --- truncated ---
```

3. Save
4. If you already have existing resources created by the operator from existing Custom Resources, you might need to force the operator to recreate the pods, by deleting the corresponding Deployment. For example:

```bash
# make sure to update CR_NAME value accordingly
$ CR_NAME=my-rhdh
$ oc delete deployment --selector app.kubernetes.io/instance="${CR_NAME}"
```

#### Metrics Add-on

For the _metrics_ add-on, we can modify the [`ama-metrics-settings-configmap`](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-settings-configmap.yaml) Config Map and enable pod annotations based scraping for the namespace your showcase instance is in. In the [example Config Map](./configuration_files/ama-metrics-settings-configmap.yaml), change the regex for the `podannotationnamespaceregex` option to match the namespaces you want to scrape from. For more information on how to configure this refer to the [official Azure docs](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration#customize-metrics-collected-by-default-targets).
Expand Down Expand Up @@ -94,7 +195,11 @@ InsightsMetrics

## Logging

Logging in backstage showcase is conducted using the [winston](https://github.com/winstonjs/winston) library. By default, logs of level `debug` are not logged. To enable debug logs, you will need to set the environment variable `LOG_LEVEL` to `debug` in your deployment in the helm chart's `values.yaml` as follows:
Logging in backstage showcase is conducted using the [winston](https://github.com/winstonjs/winston) library. By default, logs of level `debug` are not logged. To enable debug logs, you will need to set the environment variable `LOG_LEVEL` to `debug` in your deployment.

### Helm deployment

You can set the logging level by adding the environment variable in your Helm chart's `values.yaml`, as follows:

```yaml title="values.yaml"
upstream:
Expand All @@ -105,6 +210,20 @@ upstream:
value: debug
```

### Operator-backed deployment

You can set the logging level by adding the environment variable in your Custom Resource, like so:

```yaml title="env var in Custom Resource"
spec:
# Other fields omitted
application:
extraEnvs:
envs:
- name: LOG_LEVEL
value: debug
```

### Openshift Logging Integration

[Openshift Logging](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.13/html/logging/index) can be used to monitor Backstage logs. The only requirement is to correctly filter logs in Kibana. A possible filter is using the field `kubernetes.container_name` with operator `is` and value `backstage-backend`.
Expand Down

0 comments on commit 0119b18

Please sign in to comment.