Skip to content

Cross-ref relevant info about accessing monitoring dashboards in kurl/kots installs #3011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 29, 2025
Merged

Cross-ref relevant info about accessing monitoring dashboards in kurl/kots installs #3011

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bbe2523
Remove kurl info from prom monitoring topic
paigecalvert Dec 16, 2024
fab7f0b
edits
paigecalvert Dec 17, 2024
4dfa00f
add xrefs to info that applies to both installers
paigecalvert Jan 28, 2025
45896e4
add prereq for accessing dashboards
paigecalvert Jan 29, 2025
5b8ca86
Merge branch 'main' into 116046
paigecalvert Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/enterprise/monitoring-access-dashboards.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Accessing Dashboards Using Port Forwarding

This topic includes information about how to access Prometheus, Grafana, and Alertmanager in Replicated KOTS existing cluster and Replicated kURL installations.

For information about how to configure Prometheus monitoring in existing cluster installations, see [Configuring Prometheus Monitoring in Existing Cluster KOTS Installations](monitoring-applications).

## Overview

The Prometheus [expression browser](https://prometheus.io/docs/visualization/browser/), Grafana, and some preconfigured dashboards are included with Kube-Prometheus for advanced visualization. Prometheus Altertmanager is also included for alerting. You can access Prometheus, Grafana, and Alertmanager dashboards using `kubectl port-forward`.

:::note
You can also expose these pods on NodePorts or behind an ingress controller. This is an advanced use case. For information about exposing the pods on NodePorts, see [NodePorts](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/node-ports.md) in the kube-prometheus GitHub repository. For information about exposing the pods behind an ingress controller, see [Expose via Ingress](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/exposing-prometheus-alertmanager-grafana-ingress.md) in the kube-prometheus GitHub repository.
:::

## Prerequisite

For existing cluster KOTS installations, first install Prometheus in the cluster and configure monitoring. See [Configuring Prometheus Monitoring in Existing Cluster KOTS Installations](monitoring-applications)

## Access Prometheus

To access the Prometheus dashboard:

1. Run the following command to port forward the Prometheus service:

```bash
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
```

1. Access the dashboard at http://localhost:9090.

## Access Grafana

Users can access the Grafana dashboard by logging in using a default username and password. For information about configuring Grafana, see the [Grafana documentation](https://grafana.com/docs/). For information about constructing Prometheus queries, see [Querying Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/) in the Prometheus documentation.

To access the Grafana dashboard:

1. Run the following command to port forward the Grafana service:

```bash
kubectl --namespace monitoring port-forward deployment/grafana 3000
```
1. Access the dashboard at http://localhost:3000.
1. Log in to Grafana:
* **Existing cluster**: Use the default Grafana username and password: `admin:admin`.
* **kURL cluster**: The Grafana password is randomly generated by kURL and is displayed on the command line after kURL provisions the cluster. To log in, use this password generated by kURL and the username `admin`.

To retrieve the password, run the following kubectl command:

```
kubectl get secret -n monitoring grafana-admin -o jsonpath="{.data.admin-password}" | base64 -d
```

## Access Alertmanager

Alerting with Prometheus has two phases:

* Phase 1: Alerting rules in Prometheus servers send alerts to an Alertmanager.
* Phase 2: The Alertmanager then manages those alerts, including silencing, inhibition, aggregation, and sending out notifications through methods such as email, on-call notification systems, and chat platforms.

For more information about configuring Alertmanager, see [Configuration](https://prometheus.io/docs/alerting/configuration/) in the Prometheus documentation.

To access the Alertmanager dashboard:

1. Run the following command to port forward the Alertmanager service:

```
kubectl --namespace monitoring port-forward svc/prometheus-alertmanager 9093
```

1. Access the dashboard at http://localhost:9093.
100 changes: 12 additions & 88 deletions docs/enterprise/monitoring-applications.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,22 @@
import OverviewProm from "../partials/monitoring/_overview-prom.mdx"
import LimitationEc from "../partials/monitoring/_limitation-ec.mdx"

# Monitoring Applications with Prometheus
# Configuring Prometheus Monitoring in Existing Cluster KOTS Installations

This topic describes monitoring applications and clusters with Prometheus. It includes information about how to configure Prometheus monitoring for existing clusters and how to access the dashboard using a port forward.
This topic describes how to monitor applications and clusters with Prometheus in existing cluster installations with Replicated KOTS.

## Overview

<OverviewProm/>
For information about how to access Prometheus, Grafana, and Alertmanager, see [Accessing Dashboards Using Port Forwarding](/enterprise/monitoring-access-dashboards).

## Limitation
For information about consuming Prometheus metrics externally in kURL installations, see [Consuming Prometheus Metrics Externally](monitoring-external-prometheus).

<LimitationEc/>
## Overview

## Configure Monitoring in Existing Clusters {#configure-existing}
<OverviewProm/>

To configure Prometheus monitoring for applications installed in an existing cluster, connect the Admin Console to the endpoint of an installed instance of Prometheus on the cluster. See the following sections:
## Configure Prometheus Monitoring

* [Install Prometheus](#install-prometheus)
* [Connect to a Prometheus Endpoint](#connect-to-a-prometheus-endpoint)
For existing cluster installations with KOTS, users can install Prometheus in the cluster and then connect the Admin Console to the Prometheus endpoint to enable monitoring.

### Install Prometheus
### Step 1: Install Prometheus in the Cluster {#configure-existing}

Replicated recommends that you use CoreOS's Kube-Prometheus distribution for installing and configuring highly available Prometheus on an existing cluster. For more information, see the [kube-prometheus](https://github.com/coreos/kube-prometheus) GitHub repository.

Expand All @@ -43,9 +39,9 @@ To install Prometheus using the recommended Kube-Prometheus distribution:

For more information about advanced Kube-Prometheus configuration options, see [Customizing Kube-Prometheus](https://github.com/coreos/kube-prometheus#customizing-kube-prometheus) in the kube-prometheus GitHub repository.

### Connect to a Prometheus Endpoint
### Step 2: Connect to a Prometheus Endpoint

To view graphs on the Admin Console dashboard, you must provide the address of the Prometheus instance that you installed on the cluster.
To view graphs on the Admin Console dashboard, provide the address of a Prometheus instance installed in the cluster.

To connect the Admin Console to a Prometheus endpoint:

Expand All @@ -54,76 +50,4 @@ To connect the Admin Console to a Prometheus endpoint:

![Configuring Prometheus](/images/kotsadm-dashboard-configureprometheus.png)

Graphs appear on the dashboard shortly after saving the address.

## Access the Dashboards with kubectl Port Forward

You can use the commands below to access Prometheus, Grafana, and Alertmanager dashboards using `kubectl port-forward` after you install the manifests.

You can also expose these pods on NodePorts or behind an ingress controller. This is an advanced use case. For information about exposing the pods on NodePorts, see [NodePorts](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/node-ports.md) in the kube-prometheus GitHub repository. For information about exposing the pods behind an ingress controller, see [Expose via Ingress](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/exposing-prometheus-alertmanager-grafana-ingress.md) in the kube-prometheus GitHub repository.

For Replicated kURL clusters, you can consume Prometheus metrics from an external monitoring solution by connecting to the Prometheus NodePort service running in the cluster. For more information, see [Consuming Prometheus Metrics Externally](monitoring-external-prometheus).

### Access Prometheus

To access the Prometheus dashboard with a port forward:

1. Run the following command to create the port forward:

```bash
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
```

1. Access the dashboard at http://localhost:9090.

### Access Grafana

To access the Grafana dashboard with a port forward:

1. Run the following command to create the port forward:

```bash
kubectl --namespace monitoring port-forward deployment/grafana 3000
```
1. Access the dashboard at http://localhost:3000.
1. Log in to Grafana:
* **Existing cluster**: Use the default Grafana username and password: `admin:admin`.
* **kURL cluster**: The Grafana password is randomly generated by kURL and is displayed on the command line after kURL provisions the cluster. To log in, use this password generated by kURL and the username `admin`.

To retrieve the password, run the following kubectl command:

```
kubectl get secret -n monitoring grafana-admin -o jsonpath="{.data.admin-password}" | base64 -d
```

### Access Alertmanager

To access the Alertmanager dashboard with a port forward:

1. Run the following command to create the port forward:

```
kubectl --namespace monitoring port-forward svc/prometheus-alertmanager 9093
```

1. Access the dashboard at http://localhost:9093.

## About Visualizing Metrics with Grafana

In addition to the Prometheus Expression Browser, Grafana and some preconfigured dashboards are included with Kube-Prometheus for advanced visualization.

For information about configuring Grafana, see the [Grafana documentation](https://grafana.com/docs/).

For information about constructing Prometheus queries, see the [Querying Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/) in the Prometheus documentation.

For information about the Prometheus Expression Browser, see [Expression Browser](https://prometheus.io/docs/visualization/browser/) in the Prometheus documentation.


## About Alerting with Prometheus

Alerting with Prometheus has two phases:

1. Alerting rules in Prometheus servers send alerts to an Alertmanager.
1. The Alertmanager then manages those alerts, including silencing, inhibition, aggregation, and sending out notifications through methods such as email, on-call notification systems, and chat platforms.

For more information about configuring Alertmanager, see [Configuration](https://prometheus.io/docs/alerting/configuration/) in the Prometheus documentation.
Graphs appear on the dashboard shortly after saving the address.
13 changes: 10 additions & 3 deletions docs/enterprise/monitoring-external-prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import OverviewProm from "../partials/monitoring/_overview-prom.mdx"
import KurlAvailability from "../partials/kurl/_kurl-availability.mdx"

# Consuming Prometheus Metrics Externally
Expand All @@ -6,11 +7,15 @@ import KurlAvailability from "../partials/kurl/_kurl-availability.mdx"

This topic describes how to consume Prometheus metrics in Replicated kURL clusters from a monitoring service that is outside the cluster.

## About the Prometheus NodePort Service
For information about how to access Prometheus, Grafana, and Alertmanager, see [Accessing Dashboards Using Port Forwarding](/enterprise/monitoring-access-dashboards).

By default, Prometheus is included in kURL clusters as a NodePort service named `prometheus-k8s` in the `monitoring` namespace. The `prometheus-k8s` service is exposed on the IP address for each node in the cluster at port 30900.
## Overview

For more information about NodePort services, see [Type NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) in _Services_ in the Kubernetes documentation.
<OverviewProm/>

For kURL installations, if the [kURL Prometheus add-on](https://kurl.sh/docs/add-ons/prometheus) is included in the kURL installer spec, then the Prometheus monitoring system is installed alongside the application. No additional configuration is required to collect metrics and view any default and custom graphs on the Admin Console dashboard.

Prometheus is deployed in kURL clusters as a NodePort service named `prometheus-k8s` in the `monitoring` namespace. The `prometheus-k8s` service is exposed on the IP address for each node in the cluster at port 30900.

You can run the following command to view the `prometheus-k8s` service in your cluster:

Expand All @@ -25,6 +30,8 @@ prometheus-k8s NodePort 10.96.2.229 <none> 9090:30900/TCP 5hr
```
As shown in the example above, port 9090 on the `prometheus-k8s` service maps to port 30900 on each of the nodes.

For more information about NodePort services, see [Type NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) in _Services_ in the Kubernetes documentation.

## Prerequisite

Before you can consume Prometheus metrics in kURL clusters externally, ensure that firewall rules on all nodes in the cluster allow inbound TCP traffic on port 30900.
Expand Down
14 changes: 5 additions & 9 deletions docs/partials/monitoring/_overview-prom.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,15 @@
For installations in existing clusters and in Replicated kURL clusters, the Replicated KOTS Admin Console uses the open source systems monitoring tool Prometheus to collect metrics on an application and the cluster where the application is installed. Prometheus components include the main Prometheus server, which scrapes and stores time series data, an Alertmanager for alerting on metrics, and Grafana for visualizing metrics. For more information about Prometheus, see [What is Prometheus?](https://prometheus.io/docs/introduction/overview/) in the Prometheus documentation.
The KOTS Admin Console can use the open source systems monitoring tool Prometheus to collect metrics on an application and the cluster where the application is installed. Prometheus components include the main Prometheus server, which scrapes and stores time series data, an Alertmanager for alerting on metrics, and Grafana for visualizing metrics. For more information about Prometheus, see [What is Prometheus?](https://prometheus.io/docs/introduction/overview/) in the Prometheus documentation.

The Admin Console exposes graphs with key metrics collected by Prometheus in the **Monitoring** section of the dashboard. By default, the Admin Console includes the following graphs:
The Admin Console exposes graphs with key metrics collected by Prometheus in the **Monitoring** section of the dashboard. By default, the Admin Console displays the following graphs:

* Cluster disk usage
* Pod CPU usage
* Pod memory usage

In addition to these default graphs, application developers can also expose business and application level metrics and alerts on the dashboard.

The following screenshot shows an example of the **Monitoring** section on the Admin Console dashboard with the Disk Usage, CPU Usage, and Memory Usage default graphs.
The following screenshot shows an example of the **Monitoring** section on the Admin Console dashboard with the Disk Usage, CPU Usage, and Memory Usage default graphs:

<img alt="Graphs on the Admin Console dashboard" src="/images/kotsadm-dashboard-graph.png" width="600px"/>
<img alt="Graphs on the Admin Console dashboard" src="/images/kotsadm-dashboard-graph.png" width="700px"/>

[View a larger version of this image](/images/kotsadm-dashboard-graph.png)

For kURL installations, if the kURL Prometheus add-on is included in the kURL installer spec, then the Prometheus monitoring system is installed alongside the application. No additional configuration is required to collect metrics and view any default and custom graphs on the Admin Console dashboard. For more information about configuring kURL installers, see [Creating a kURL Installer](/vendor/packaging-embedded-kubernetes).

For existing cluster installations with KOTS, users can install Prometheus and connect the Admin Console to the Prometheus endpoint to enable monitoring. For information about how to set up Prometheus monitoring in existing clusters, see [Configure Monitoring in Existing Clusters](/enterprise/monitoring-applications#configure-existing).
[View a larger version of this image](/images/kotsadm-dashboard-graph.png)
9 changes: 8 additions & 1 deletion sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,14 @@ const sidebars = {
'enterprise/auth-configuring-rbac',
],
},
'enterprise/monitoring-applications',
{
type: 'category',
label: 'Monitoring Applications with Prometheus',
items: [
'enterprise/monitoring-applications',
'enterprise/monitoring-access-dashboards',
],
},
'enterprise/status-viewing-details',
'enterprise/delete-admin-console',
{
Expand Down