Setting up Monitoring with Prometheus and Grafana

Setting Up Monitoring Infrastructure with Node Exporter, Prometheus, and Grafana

Introduction

This guide provides step-by-step instructions for setting up a comprehensive monitoring infrastructure using Node Exporter, Prometheus, and Grafana. This setup is designed to enable efficient system metric collection, storage, and visualization, with a focus on accommodating the needs of a Python DevOps team.

Key components covered in this documentation:

• Node Exporter: For collecting and exposing system metrics
• Prometheus: For scraping and storing time-series data
• Grafana: For visualizing metrics and setting up alerts

The guide is structured to walk you through the installation and configuration of each component, including:

• Detailed installation steps for each tool
• Creation of necessary system users and directories
• Configuration of systemd service files for automatic startup
• Basic setup of Prometheus to scrape Node Exporter metrics
• Initial Grafana configuration

Additionally, this document includes a section on Grafana user management and access control, ensuring that you can properly manage user access and permissions within your monitoring setup.

By following this guide, you'll establish a robust monitoring system capable of tracking system performance, visualizing key metrics, and setting up alerts for your DevOps environment.

Node exporter Installation

Lets start with the installation of the node exporter

1. Download and Install Node exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvfz node_exporter-1.8.2.linux-amd64.tar.gz
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin
rm -rf node_exporter-1.8.2.linux-amd64.tar.gz

2. Create a system user for Node Exporter:

sudo useradd -rs /bin/false node_exporter

The -rs flags in the useradd command serve the following purposes:

-r: This option adds a system account. System accounts are typically used for services and have user IDs lower than 1000. -s /bin/false: This sets the user's login shell to /bin/false, preventing the user from logging in interactively.

3. Create a systemd service file

sudo nano /etc/systemd/system/node_exporter.service

Add the following contents to the service file:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Start and enable the systemd service node_exporter.service

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Now that the node exporter has been installed, we will set up prometheus and grafana for the python-devops team: Prometheus setup

1. Switch to the python-devops account:

sudo su - python-devops

2. Install Prometheus for the Python Devops team:

• Download and Install prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz
tar xvfz prometheus-2.53.1.linux-amd64.tar.gz
sudo mv prometheus-2.53.1.linux-amd64 /opt/prometheus-python
rm -rf prometheus-2.53.1.linux-amd64.tar.gz

• Create a system user for Prometheus

sudo useradd -rs /bin/false prometheus

• Create necessary directories and set permissions

sudo mkdir /etc/prometheus /var/lib/prometheus-python
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus-python
sudo chown -R prometheus:prometheus /opt/prometheus-python/

The command sudo mkdir /etc/prometheus /var/lib/prometheus creates two directories:

/etc/prometheus: This is typically used for configuration files related to Prometheus. This is where your Prometheus configuration file (prometheus.yml) is placed.

/var/lib/prometheus: This directory is generally used for storing Prometheus data, such as time series data.

The command sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus-python changes the ownership of the directories. This ensures that the prometheus user has the necessary permissions to read and write to these directories, which is essential for Prometheus to operate correctly when running under this specific user.

• Finally, create a configuration file in the etc/prometheus

sudo nano /etc/prometheus/prometheus.yml

Add the following to the configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

This configuration tells Prometheus scrape metrics from the Node Exporter instance on localhost:9091 every 15 seconds and also store the scraped metrics in the TSDB (time-series database) for 30 days.

3. Create a systemd service file for the prometheus-python.service:

sudo nano /etc/systemd/system/prometheus-python.service

Add the following content:

[Unit]
Description=Prometheus for Python DevOps Team
After=network.target

[Service]
User=prometheus-python
Group=prometheus-python
Type=simple
ExecStart=/opt/prometheus-python/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus-python/ \
  --web.console.templates=/opt/prometheus-python/consoles \
  --web.console.libraries=/opt/prometheus-python/console_libraries \
  --web.listen-address=:9091
  --storage.tsdb.retention.time=1y
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This systemd service file that configures a Prometheus instance for the Python DevOps Team.

Service Description

• Starts a Prometheus instance specifically for the Python DevOps Team.
  Service Configuration
• Runs as the prometheususer and group.
• Uses the prometheus.yml configuration file located in /etc/prometheus/.
• Stores metrics in /var/lib/prometheus-python/.
• Serves the Prometheus web console on port 9091.
  Service Management
• Starts the service after the network target is reached.
• Enables the service to be started automatically on boot. The line WantedBy=multi-user.target in the [Install] section enables the service to be started automatically on boot.

Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start prometheus-python
sudo systemctl enable prometheus-python

Grafana setup Install Grafana for the python-devops team

1. Download and Install Grafana

wget https://dl.grafana.com/oss/release/grafana-11.1.3.linux-amd64.tar.gz
tar -zxvf grafana-11.1.3.linux-amd64.tar.gz
sudo mv grafana-v11.1.3 /opt/grafana-python
rm -rf grafana-11.1.3.linux-amd64.tar.gz

2. Create a system user for Grafana

sudo useradd -rs /bin/false grafana-python

3. Create necessary directories and set permissions

sudo mkdir /etc/grafana-python /var/lib/grafana-python
sudo chown grafana-python:grafana-python /etc/grafana-python /var/lib/grafana-python

4. Create a configuration file

sudo nano /etc/grafana-python/grafana.ini

5. Add the following content to the configuration file:

[server]
http_port = 3001

[paths]
data = /var/lib/grafana-python
logs = /var/log/grafana-python
plugins = /var/lib/grafana-python/plugins

6. Create a systemd service file:

sudo nano /etc/systemd/system/grafana-python.service

7. Add the following content:

[Unit]
Description=Grafana for Python DevOps Team
After=network.target

[Service]
User=grafana-python
Group=grafana-python
Type=simple
ExecStart=/opt/grafana-python/bin/grafana-server \
  -config /etc/grafana-python/grafana.ini \
  -homepath /opt/grafana-python
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

8. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start grafana-python
sudo systemctl enable grafana-python

Grafana User Management and Access Control

Creating Users

1. Navigate to "Server Admin" > "Users"
2. Click "Invite User" or "New User"
3. Fill in required details:
   • Name
   • Email
   • Username (if "New User")
   • Password (if "New User")
4. Set organization and role
5. Click "Submit"

Access Control

Roles

• Admin: Full access to all organizations
• Editor: Can edit and create dashboards
• Viewer: Can view dashboards

Organization Roles

• Admin: Manage users and permissions within the organization
• Editor: Edit and create dashboards
• Viewer: View dashboards

Teams

1. Go to "Configuration" > "Teams"
2. Click "New Team"
3. Add members and set permissions

Folder Permissions

1. Navigate to the desired folder
2. Click the gear icon > "Permissions"
3. Add users, teams, or roles
4. Set appropriate access levels

Dashboard Permissions

1. Open the dashboard
2. Click the gear icon > "Permissions"
3. Add users, teams, or roles
4. Set viewer, editor, or admin access