diff --git a/.gitignore b/.gitignore index b63db6c..66019d4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,8 @@ +# Custom ignores +observability/examples/simple/observability-simple + + +# Python ignores # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] diff --git a/observability/docs/get-started/quickstart-manual.md b/observability/docs/get-started/quickstart-manual.md new file mode 100644 index 0000000..71d009c --- /dev/null +++ b/observability/docs/get-started/quickstart-manual.md @@ -0,0 +1,36 @@ + +## Manual Quickstart + +The quickstart page uses a script to setup the folders for you. + +This page instead details how to do it manually, to provide clarity. + +## Step 1: Setup directory +Create the necessary directory structure +``` +mkdir -p observability-simple/prometheus/scrape-configs/probers +mkdir -p observability-simple/prometheus/scrape-configs/exporters +``` + + +Download these two files from github, and place in the right folder + +- [docker-compose.yml](https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/docker-compose.yml) in observability-simple/ +- [prometheus/scrape-configs/probers/probe-simple.yml](https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml) + +### Step 2: Start the stack + +In the folder containing the downloaded files, run: + +```bash +docker compose up -d +``` + +### Step 3: Access the dashboards +Open your web browser and go to: + +`localhost/grafana` + +You should see the Grafana dashboard displaying the availability of the sample web page. + +There it is, you can now see the availability of the cogstack homepage, as well as the observability stack diff --git a/observability/docs/get-started/quickstart.md b/observability/docs/get-started/quickstart.md new file mode 100644 index 0000000..565ff5c --- /dev/null +++ b/observability/docs/get-started/quickstart.md @@ -0,0 +1,61 @@ +## QuickStart + +This tutorial guides you through running the simplest setup of the observability stack using example configuration files and Docker Compose. + +After completing these steps, you will have a full observability stack running locally, showing the availability of web pages you want to target + +### Requirements + +- Docker installed ([install Docker](https://docs.docker.com/get-docker/)) +- Docker Compose installed ([install Docker Compose](https://docs.docker.com/compose/install/)) +- A terminal with network access + +### Step 1: Run the Quickstart script + +Run this quickstart script to setup the project +```bash +curl https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/quickstart.sh | bash +``` +Now go to "http://localhost/grafana" to see the dashboards + +Thats everything. The stack is running and you can see the availability. + + +### Optional Step: Probe your own web page +Now you can look at getting monitoring of your own page + +In your current folder, edit the file `prometheus/scrape-configs/probers/probe-simple.yml` that you downloaded from git. + +Add the following yml to the bottom of the file: + +```yaml +- targets: + - https://google.com + labels: + name: google + job: probe-my-own-site +``` + + +The change should get applied automatically, but if you dont want to wait then run +``` +docker compose restart +``` + +Now refresh the grafana dashboard, and you can see the availability of google.com, it's probably 100%! + + +## Next steps +This is the end of this quickstart tutorial, that enables probing availability of endpoints. + +For the next steps we can: +- Productionise our deployment to enable further features +- Enable *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters +- Enable *Alerting* based on our availability and a defined Service Level Objective (SLO) +- Look further into the available dashboards +- Fully customize the stack with our own dashboards, recording rules and metrics + + + + + diff --git a/observability/docs/get-started/setup-tutorial.md b/observability/docs/get-started/setup-tutorial.md new file mode 100644 index 0000000..8748972 --- /dev/null +++ b/observability/docs/get-started/setup-tutorial.md @@ -0,0 +1,17 @@ + +This page + + + +## Run the Observability Stack using Docker Compose + +See the /examples folder for a working example of running this + +To setup the stack for your deployment: +- Create prometheus configurations as listed below +- Copy this docker-compose.yml file +- Mount your site config files into `/etc/prometheus/cogstack/site` +- Run with docker compose + +To collect metrics from VMs to fill out the dashboards +- Run the Exporters on each VM as detailed below \ No newline at end of file diff --git a/observability/docs/get-started/userguide-tutorial.md b/observability/docs/get-started/userguide-tutorial.md new file mode 100644 index 0000000..e69de29 diff --git a/observability/docs/overview.md b/observability/docs/overview.md new file mode 100644 index 0000000..84158f2 --- /dev/null +++ b/observability/docs/overview.md @@ -0,0 +1,18 @@ +# Cogstack Observability Stack + +This project provides observability of a cogstack deployment. + +It provides the following features: +- Dashboards for availability +- Telemetry of infrastructure such as host memory usage, and elasticsearch index size +- Alerting based on a defined Service Level Objective (SLO) and burn rates +- Blackbox Probing of services to find service level indicators of uptime and latency +- A working inventory of what is running where + +## Contents + +See the [Quickstart](./get-started/quickstart.md) to see how to easily run this stack. + + + + diff --git a/observability/examples/simple/docker-compose.yml b/observability/examples/simple/docker-compose.yml index 8be8a9f..659a067 100755 --- a/observability/examples/simple/docker-compose.yml +++ b/observability/examples/simple/docker-compose.yml @@ -6,7 +6,7 @@ services: image: cogstacksystems/cogstack-observability-prometheus:latest restart: unless-stopped volumes: - - ./prometheus:/etc/prometheus/cogstack/site + - ./prometheus:/etc/prometheus/cogstack/site/ - prometheus-data:/prometheus networks: - observability @@ -17,8 +17,6 @@ services: - grafana-data:/var/lib/grafana networks: - observability - ports: - - '3000:3000' environment: - GF_AUTH_ANONYMOUS_ENABLED=true # Allows use of grafana without sign in - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer @@ -29,16 +27,23 @@ services: restart: unless-stopped ports: - "80:80" - - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro # So that Traefik can listen to the Docker events blackbox-exporter: image: cogstacksystems/cogstack-observability-blackbox-exporter:latest restart: unless-stopped - ports: - - "9115:9115" networks: - - observability-exporters + - observability + node-exporter: + image: prom/node-exporter + restart: unless-stopped + networks: + - observability + labels: + - "traefik.enable=true" + - "traefik.http.routers.node-exporter.rule=PathPrefix(`/node-exporter`)" + - "traefik.http.middlewares.node-exporter-stripprefix.stripprefix.prefixes=/node-exporter" + - "traefik.http.routers.node-exporter.middlewares=node-exporter-stripprefix@docker" networks: observability: driver: bridge diff --git a/observability/examples/simple/prometheus/scrape-configs/exporters/exporters-simple.yml b/observability/examples/simple/prometheus/scrape-configs/exporters/exporters-simple.yml new file mode 100644 index 0000000..8036d08 --- /dev/null +++ b/observability/examples/simple/prometheus/scrape-configs/exporters/exporters-simple.yml @@ -0,0 +1,5 @@ +- targets: + - cogstack-observability-node-exporter-1:9100 + labels: + job: node_exporter + host: localhost \ No newline at end of file diff --git a/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml b/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml index a350490..b3d7353 100644 --- a/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml +++ b/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml @@ -3,4 +3,4 @@ - https://cogstack.org labels: name: cogstack-homepage - job: "probe-simple-example" \ No newline at end of file + job: probe-services \ No newline at end of file diff --git a/observability/examples/simple/quickstart.sh b/observability/examples/simple/quickstart.sh new file mode 100644 index 0000000..45b67b7 --- /dev/null +++ b/observability/examples/simple/quickstart.sh @@ -0,0 +1,26 @@ +#!/bin/bash +set -e + +mkdir -p observability-simple/prometheus/scrape-configs/probers +mkdir -p observability-simple/prometheus/scrape-configs/exporters +cd observability-simple + +echo "Downloading docker-compose.yml..." +curl -fsSL -o docker-compose.yml \ + https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/docker-compose.yml + +echo "Downloading probe-simple.yml into prometheus/scrape-configs/probers/..." +curl -fsSL -o prometheus/scrape-configs/probers/probe-simple.yml \ + https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/prometheus/scrape-configs/probers/probe-simple.yml + +echo "Downloading exporters-simple.yml into prometheus/scrape-configs/exporters/..." +curl -fsSL -o prometheus/scrape-configs/exporters/exporters-simple.yml \ + https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/prometheus/scrape-configs/exporters/exporters-simple.yaml + + +echo "Setup complete in observability-simple/" + +echo "Starting the observability stack" +docker compose up -d + + echo "Please open http://localhost/grafana in your browser" \ No newline at end of file