Improve Grafana Alloy useability

docs/observability/customization/custom-prometheus-configs.md

@@ -12,7 +12,7 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
 
 
 ### Add Exporters to Prometheus
-- `prometheus/scrape-configs/exporters/` -
+- `prometheus/scrape-configs/exporters/`
  Add yaml files into this folder. These file should contain all exporter prometheus metrics, for example from node_exporter or CAdvisor. Add any hosts and ip addresses you want to collect /metrics from will be retrieved
 
 ```yaml 
@@ -26,6 +26,9 @@ Mount these files under `site/prometheus/scrape-configs/exporters/*.yml` in dock
     # __metrics_path__: /path/metrics  # Optionally override the metrics path, the default is just /metrics
 # ... add all targets
 ```
+
+Note that this project is setup to run Grafana Alloy to push metrics from individual VMs. The config in `prometheus/scrape-configs/exporters/` is an alternative way, allowing you to pull metrics from any other services.
+
 ## Custom Prometheus Scrape Configs
 
 You can add compeltely custom prometheus scrape configs and recording rules by mounting in docker.

docs/observability/get-started/quickstart.md

@@ -28,7 +28,7 @@ If you can't use the script, see the [Manual Quickstart](../reference/quickstart
 ### Optional Step: Probe your own web page
 Now you can look at getting monitoring of your own page
 
-1. In your current folder, in the file `prometheus/scrape-configs/probers/probe-simple.yml` add the following yml to the bottom of the file:
+1. In your current folder, in the file `alloy/probers/probe-observability.yml` add the following yml to the bottom of the file:
 
 ```yaml
 - targets:
@@ -52,11 +52,7 @@ This is the end of this quickstart tutorial, that enables probing availability o
 
 For the next steps we can:
 - Look deeper into the observability dashboards, on [Dashboards Userguide](./userguide-tutorial.md)
-- Productionise our deployment to enable further features
-- Configure *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
-- Enable *Alerting* based on our availability and a defined Service Level Objective (SLO)
-- Setup further *Probing* of our running services to get availability metrics
-- Fully customize the stack with our own dashboards, recording rules and metrics
+- Productionise our deployment to enable further features  by following [Production Setup](../setup/production-setup.md)
 
 
 

docs/observability/setup/production-setup.md

@@ -6,6 +6,12 @@ If you're new, we recommend completing the [Quickstart Tutorial](../get-started/
 
 By the end of the tutorial, you will have a complete stack offering all the observability features, customized to your usage. 
 
+We will run the stack and then:
+- Configure *Telemetry* like VM memory usage, and Elasticsearch index size, by running Exporters
+- Enable *Alerting* based on our availability and a defined Service Level Objective (SLO)
+- Setup further *Probing* of our running services to get availability metrics
+
+
 ---
 
 ## Step 1: Understand the Folder Structure
@@ -44,7 +50,7 @@ Downloads the example docker compose files:
 
 Downloads the configurations:
 - [alloy/probers/probe-external.yml](../../../observability/examples/full/alloy/probers/probe-external.yml)
-- [alloy/probers/probe-internal.yml ](../../../observability/examples/full/alloy/probers/probe-internal.yml)
+- [alloy/probers/probe-observability.yml ](../../../observability/examples/full/alloy/probers/probe-observability.yml)
 - [prometheus/scrape-configs/exporters/exporters.yml](../../../observability/examples/full/prometheus/scrape-configs/exporters/exporters.yml)
 - [prometheus/scrape-configs/recording-rules/slo.yml](../../../observability/examples/full/prometheus/scrape-configs/recording-rules/slo.yml)
 
@@ -58,10 +64,10 @@ The files come with basic defaults, so we can now run the stack
 
    ```
    docker compose up -d
-   docker compose -f exporters.docker-compose.yml up -d
    ```
 
-This will launch Prometheus, Grafana, and all required services with 
+This will launch Prometheus, Grafana, and Alloy
+
 
 
 ## Step 4: Create Site-Specific Config Files
@@ -70,21 +76,24 @@ You must provide your own scrape and recording rules to tell Prometheus what to
 This is probably the hardest step: You will actually need to know what is running, and where it is! Building out these config files will give you that inventory, and give a real definition of what is running where.
 
 - Probers: HTTP endpoints you want to monitor for availability
-  - Add files in `scrape-configs/probers/*.yml`
+  - Add files in `alloy/probers/*.yml`
   - [Configure Probers](./probing.md)
-
-- Exporters: Targets like Elasticsearch or Docker
-  - Add files in `scrape-configs/exporters/*.yml`
-  - [Add Exporters](./telemetry.md)
+
+- Telemetry: Run Grafana Alloy on every VM you want telemetry from
+    - [Configure Telemetry](./probing.md)
 
 - Recording Rules: Define uptime goals or custom aggregations
   - Add files in `recording-rules/*.yml`
   - [Enable Alerting](./alerting.md)
 
-## Step 5: Run Exporters Everywhere
-The exporters need to be run on each VM that you want information from. It's a pull model, not push.
+## Step 5: Run Grafana Alloy on every VM
+The Grafana Alloy image needs to be run on each VM that you want to get information from. 
 
+Use the example docker compose file in [exporters.docker-compose.yml](../../../observability/examples/full/exporters.docker-compose.yml) which will start up alloy and get metrics
 
+   ```
+   docker compose -f exporters.docker-compose.yml up -d
+   ```
 ---
 
 ## What’s Next?
@@ -93,7 +102,7 @@ Your observability stack is now monitoring your services, and you have a product
 
 You can now setup prometheus with any telemetry or probers required following the remaining steps in [Setup](./_index.md)
 
-For the last steps, you can 
+For the last steps, you can:
 
 - Run the exporters on all the VMs that you want access to 
 - Deploy the stack in produciton

docs/observability/setup/telemetry.md

@@ -10,7 +10,6 @@ Grafana Alloy is used to get telemetry. These features are configured by default
 - Elastic Search Exporter: Get ES metrics like index size
 - CAdvisor: This gives docker metrics, eg what containers are running
 
-
 ## How to get Telemetry
 
 - Copy this docker compose file: (exporters.docker-compose.yml)[observability/examples/full/exporters.docker-compose.yml]

observability/examples/full/alloy/probers/probe-observability.yml

@@ -0,0 +1,12 @@
+- targets:
+    - cogstack-observability-traefik-1/grafana/api/health
+  labels:
+    name: grafana
+    job: probe-observability-stack
+    host: localhost
+- targets:
+    - cogstack-observability-traefik-1/prometheus/-/healthy
+  labels:
+    name: prometheus
+    job: probe-observability-stack
+    host: localhost

observability/examples/full/docker-compose.yml

@@ -6,6 +6,8 @@ services:
     image: cogstacksystems/cogstack-observability-alloy:latest
     ports:
       - "12345:12345"
+    networks:
+      - observability
     volumes:
       - ${BASE_DIR-.}/alloy/probers:/etc/alloy/probers
       # CAdvisor

observability/examples/full/exporters.elastic.docker-compose.yml

@@ -15,7 +15,7 @@ services:
       - PROMETHEUS_URL=http://cogstack-observability-prometheus-1:9090/prometheus
       - ALLOY_HOSTNAME=${ALLOY_HOSTNAME-localhost} # Used to add a label to metrics
       - ALLOY_IP_ADDRESS=${ALLOY_IP_ADDRESS-localhost} # Used to add a label to metrics
-      - ELASTICSEARCH_URL=${ELASTICSEARCH_URL-https://elassticsearch-1:9200}
+      - ELASTICSEARCH_URL=${ELASTICSEARCH_URL-https://elasticsearch-1:9200}
       - ELASTICSEARCH_USERNAME=${ELASTICSEARCH_USERNAME-user} # Used to get metrics from Elasticsearch
       - ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD-pass} # Used to get metrics from Elasticsearch
     networks:

observability/examples/full/full-quickstart.sh

@@ -10,6 +10,7 @@ download_to() {
   curl -fsSL -o "$path" "$url"
 }
 
+mkdir -p cogstack-observability/alloy/probers
 mkdir -p cogstack-observability/prometheus/scrape-configs/probers
 mkdir -p cogstack-observability/prometheus/scrape-configs/exporters
 mkdir -p cogstack-observability/prometheus/scrape-configs/recording-rules

observability/examples/simple/alloy/probers/probe-observability.yml

@@ -0,0 +1,12 @@
+- targets:
+    - cogstack-observability-traefik-1/grafana/api/health
+  labels:
+    name: grafana
+    job: probe-observability-stack
+    host: localhost
+- targets:
+    - cogstack-observability-traefik-1/prometheus/-/healthy
+  labels:
+    name: prometheus
+    job: probe-observability-stack
+    host: localhost

observability/examples/simple/quickstart.sh

@@ -9,8 +9,8 @@ curl -fsSL -o docker-compose.yml \
   https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/docker-compose.yml
 
 echo "Downloading probe-simple.yml into alloy/probers/..."
-curl -fsSL -o probers/probe-simple.yml \
-  https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/probers/probe-simple.yml
+curl -fsSL -o probers/probe-observability.yml \
+  https://raw.githubusercontent.com/CogStack/cogstack-platform-toolkit/main/observability/examples/simple/probers/probe-observability.yml
 
 echo "Setup complete in observability-simple/"
 

observability/grafana-alloy/Dockerfile

@@ -3,6 +3,8 @@ FROM grafana/alloy:latest
 LABEL traefik.enable="true" \
     traefik.http.routers.alloy.rule="PathPrefix(`/alloy`)" 
 
+RUN mkdir -p /etc/alloy/probers
+
 COPY ./defaults /etc/alloy
 
 CMD [ \

observability/grafana-alloy/defaults/config.alloy

@@ -8,31 +8,7 @@ prometheus.remote_write "default" {
     url = sys.env("PROMETHEUS_URL") + "/api/v1/write"
   }
   external_labels = {
-    host = sys.env("ALLOY_HOSTNAME"),
-    ip_address = sys.env("ALLOY_IP_ADDRESS"),
+    alloy_hostname = sys.env("ALLOY_HOSTNAME"),
+    alloy_ip_address = sys.env("ALLOY_IP_ADDRESS"),
   } 
-}
-
-prometheus.scrape "exporter" {
-    scrape_interval = "15s"
-    targets    = array.concat(
-        prometheus.exporter.self.alloy.targets,
-        prometheus.exporter.cadvisor.local_cadvisor.targets,
-        prometheus.exporter.unix.local_node_exporter.targets,
-    )
-    forward_to = [prometheus.remote_write.default.receiver]
-}
-
-// Alloys internal metrics
-prometheus.exporter.self "alloy" {
-}
-
-// CAdvisor
-prometheus.exporter.cadvisor "local_cadvisor" {
-  docker_host = "unix:///var/run/docker.sock"
-  storage_duration = "5m"
-}
-
-// Node exporter
-prometheus.exporter.unix "local_node_exporter" {
-}
+}