This application collects custom formatted log from nginx via Syslog, counts all the data and exports metrics to Prometheus server.
Switch branches/tags
Nothing to show
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
contrib add 'Tag' variable to grafana dashboard Dec 4, 2018
spec add some specs Jul 30, 2018
src use systemd notifications Dec 4, 2018
.gitignore Initial revision Jul 30, 2018
LICENSE Initial revision Jul 30, 2018
Makefile add build_run target Jul 30, 2018
README.md add 'access_log ... if=...' example Dec 4, 2018
TODO.md add TODO Jul 31, 2018
shard.lock bump Dec 3, 2018
shard.yml bump Dec 3, 2018

README.md

Description

This application collects custom formatted log from nginx via Syslog, counts all the data and exports metrics to Prometheus server.

┌───────┐                     ┌─────────────────────────────────┐
│ nginx │───[ syslog/UDP ]───▸│ nginx-prometheus-shiny-exporter │
└───────┘                     └─────────────────────────────────┘
                                              ▴
                                              ║
                                           [ HTTP ]
                                              ║
                                              ▾
                                       ┌────────────┐
                                       │ Prometheus │
                                       └────────────┘

It works similar to nginx_request_exporter but written in Crystal.

Status

master branch contains a production ready version. It is able to return a limited number of metrics. More ones will be added in future.

I use this application on Russian National Platform for Open Education https://openedu.ru

Installation

  1. Install Crystal
  2. apt install build-essential pkg-config zlib1g-dev libssl-dev
  3. make build

Usage

Create a virtual host:

log_format collect_status 'stts|$host|$status';
log_format collect_time   'reqt|$host|$request_time';

map $request_uri $is_not_static {
  default        1;
  "~^/media/"    0;
  "~^/static/"   0;
}

server {
  listen 127.0.0.1:8877 default_server;

  access_log syslog:server=127.0.0.1:9467,tag=default collect_status;
  access_log syslog:server=127.0.0.1:9467,tag=default collect_time;
  access_log syslog:server=127.0.0.1:9467,tag=app collect_status if=$is_not_static;
  access_log syslog:server=127.0.0.1:9467,tag=app collect_time   if=$is_not_static;

  location / {
    # …
  }

  location /special/ {
    access_log syslog:server=127.0.0.1:9467,tag=special collect_status;
    access_log syslog:server=127.0.0.1:9467,tag=special collect_time;
    # …
  }
}

Do several requests and check the metrics:

$ curl -s http://localhost:9467/metrics
# HELP nginx_request_status A metric
# TYPE nginx_request_status counter
nginx_request_status{host="127.0.0.1",tag="default",status="404"} 2
nginx_request_status{host="127.0.0.1",tag="default",status="499"} 3
nginx_request_status{host="127.0.0.1",tag="default",status="503"} 2
nginx_request_status{host="127.0.0.1",tag="default",status="204"} 1
nginx_request_status_ranges{host="127.0.0.1",tag="default",range="100-399"} 1
nginx_request_status_ranges{host="127.0.0.1",tag="default",range="400-498"} 2
nginx_request_status_ranges{host="127.0.0.1",tag="default",range="499"} 3
nginx_request_status_ranges{host="127.0.0.1",tag="default",range="500-599"} 2
# HELP nginx_request_time A metric
# TYPE nginx_request_time counter
nginx_request_time_sum{host="127.0.0.1",tag="default"} 0.0
nginx_request_time_count{host="127.0.0.1",tag="default"} 8

Add to prometheus a block like this:

scrape_configs:
  - job_name: nginx
    scrape_interval: 5s
    scrape_timeout:  5s
    static_configs:
      - targets:
          - 192.0.2.8:9467

If you use Grafana, a sample dashboard can be found in contrib/ directory of the project.

FAQ

Q: Why shiny?
A: Because it is writen in Crystal