Setting up Monitoring with Prometheus and Grafana

We will be using node exporter to scrape system metrics and grafana for visualization.

Node exporter Installation

Lets start with the installation of the node exporter

1. Download and Install Node exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvfz node_exporter-1.8.2.linux-amd64.tar.gz
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin
rm -rf node_exporter-1.8.2.linux-amd64.tar.gz

2. Create a system user for Node Exporter:

sudo useradd -rs /bin/false node_exporter

The -rs flags in the useradd command serve the following purposes:

-r: This option adds a system account. System accounts are typically used for services and have user IDs lower than 1000. -s /bin/false: This sets the user's login shell to /bin/false, preventing the user from logging in interactively.

3. Create a systemd service file

sudo nano /etc/systemd/system/node_exporter.service

Add the following contents to the service file:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Start and enable the systemd service node_exporter.service

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Now that the node exporter has been installed, we will set up prometheus and grafana for the python-devops team: Prometheus setup

1. Switch to the python-devops account:

sudo su - python-devops

2. Install Prometheus for the Python Devops team:

• Download and Install prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz
tar xvfz prometheus-2.53.1.linux-amd64.tar.gz
sudo mv prometheus-2.53.1.linux-amd64 /opt/prometheus-python
rm -rf prometheus-2.53.1.linux-amd64.tar.gz

• Create a system user for Prometheus

sudo useradd -rs /bin/false prometheus-python

• Create necessary directories and set permissions

sudo mkdir /etc/prometheus-python /var/lib/prometheus-python
sudo chown prometheus-python:prometheus-python /etc/prometheus-python /var/lib/prometheus-python
sudo chown -R prometheus-python:prometheus-python /opt/prometheus-python/

The command sudo mkdir /etc/prometheus-python /var/lib/prometheus-python creates two directories:

/etc/prometheus-python: This is typically used for configuration files related to Prometheus. This is where your Prometheus configuration file (prometheus.yml) is placed.

/var/lib/prometheus-python: This directory is generally used for storing Prometheus data, such as time series data.

The command sudo chown prometheus-python:prometheus-python /etc/prometheus-python /var/lib/prometheus-python changes the ownership of the directories. This ensures that the prometheus-python user has the necessary permissions to read and write to these directories, which is essential for Prometheus to operate correctly when running under this specific user.

• Finally, create a configuration file in the etc/prometheus-python

sudo nano /etc/prometheus-python/prometheus.yml

Add the following to the configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

This configuration tells Prometheus scrape metrics from the Node Exporter instance on localhost:9091 every 15 seconds and also store the scraped metrics in the TSDB (time-series database) for 30 days.

3. Create a systemd service file for the prometheus-python.service:

sudo nano /etc/systemd/system/prometheus-python.service

Add the following content:

[Unit]
Description=Prometheus for Python DevOps Team
After=network.target

[Service]
User=prometheus-python
Group=prometheus-python
Type=simple
ExecStart=/opt/prometheus-python/prometheus \
  --config.file /etc/prometheus-python/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus-python/ \
  --web.console.templates=/opt/prometheus-python/consoles \
  --web.console.libraries=/opt/prometheus-python/console_libraries \
  --web.listen-address=:9091
  --storage.tsdb.retention.time=1y
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This systemd service file that configures a Prometheus instance for the Python DevOps Team.

Service Description

• Starts a Prometheus instance specifically for the Python DevOps Team.
  Service Configuration
• Runs as the prometheus-python user and group.
• Uses the prometheus.yml configuration file located in /etc/prometheus-python/.
• Stores metrics in /var/lib/prometheus-python/.
• Serves the Prometheus web console on port 9091.
  Service Management
• Starts the service after the network target is reached.
• Enables the service to be started automatically on boot. The line WantedBy=multi-user.target in the [Install] section enables the service to be started automatically on boot.

Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start prometheus-python
sudo systemctl enable prometheus-python

Grafana setup Install Grafana for the python-devops team

1. Download and Install Grafana

wget https://dl.grafana.com/oss/release/grafana-11.1.3.linux-amd64.tar.gz
tar -zxvf grafana-11.1.3.linux-amd64.tar.gz
sudo mv grafana-v11.1.3 /opt/grafana-python
rm -rf grafana-11.1.3.linux-amd64.tar.gz

2. Create a system user for Grafana

sudo useradd -rs /bin/false grafana-python

3. Create necessary directories and set permissions

sudo mkdir /etc/grafana-python /var/lib/grafana-python
sudo chown grafana-python:grafana-python /etc/grafana-python /var/lib/grafana-python

4. Create a configuration file

sudo nano /etc/grafana-python/grafana.ini

5. Add the following content to the configuration file:

[server]
http_port = 3001

[paths]
data = /var/lib/grafana-python
logs = /var/log/grafana-python
plugins = /var/lib/grafana-python/plugins

6. Create a systemd service file:

sudo nano /etc/systemd/system/grafana-python.service

7. Add the following content:

[Unit]
Description=Grafana for Python DevOps Team
After=network.target

[Service]
User=grafana-python
Group=grafana-python
Type=simple
ExecStart=/opt/grafana-python/bin/grafana-server \
  -config /etc/grafana-python/grafana.ini \
  -homepath /opt/grafana-python
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

8. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start grafana-python
sudo systemctl enable grafana-python