Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus server, exporter and altermanager roles #1310

Open
wants to merge 33 commits into
base: master
Choose a base branch
from

Conversation

pedroluislopez
Copy link
Member

@pedroluislopez pedroluislopez commented Mar 30, 2020

  • Prometheus server is configured with file discovery by default.
  • Prometheus exporter can be configure and install exporter debian packages, exporter releases from repo projects and can configure built-in service exporters (like rabbitmq server 3.8).
  • Include prometheus alertmanager role.
  • Prometheus exporter and alertmanager configure file discovery files in server.
  • Allow customize exporters startup variables.
  • PKI: Support TLS and cert authentication with Ghostunnel https://github.com/square/ghostunnel. between prometheus server and exporters and alertmanagers.
  • Configured Nginx basic auth for Prometheus server front and alertmanager front.
  • Support for prometheus rules.

Missing:

  • Rules files.
  • Prometheus alertmanager role.
  • HTTP reverse proxy for SSL and auth. In prometheus recommend https://github.com/square/ghostunnel for production use. Maybe it's a better option than installing nginx.
  • Write documentation.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…access to sockets

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…ault configuratio files.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@reixd
Copy link
Contributor

reixd commented Apr 14, 2020

Any news on this. I really would like to have Prom working as a role

@ganto
Copy link
Contributor

ganto commented Apr 15, 2020

I was quickly trying to compare this PR with #160 which also tries to add Prometheus support to DebOps:

In your code, I very like the facts-based integration of the exporters with the Prometheus server.

I'm currently running a setup based on the code in #160 and use a few features that are missing here. Especially these are:

  • Alertmanager integration
  • Nginx reverse proxy for SSL and basic auth in front of exporters and Prometheus server

These features are imho essential (at least for my setup).

What I'm still missing in both PR is the customization of startup arguments for the exporters and general customization of exporters. E.g. I'm running a systemd timer to provide data via node exporter collector.textfile.directory.

The Ansible code looks decent. However, I personally don't like the include_role tasks. I don't know if this changed recently but in the past DebOps wouldn't do this and use dependencies in the meta file instead.

@pedroluislopez pedroluislopez changed the title Prometheus server and prometheus exporter roles [WIP] Prometheus server and prometheus exporter roles Apr 15, 2020
@pedroluislopez
Copy link
Member Author

pedroluislopez commented Apr 15, 2020

I've set the PR in WIP.

So, missing features:

  • Rules files.
  • Prometheus alertmanager role.
  • HTTP reverse proxy for SSL and auth. In prometheus recommend https://github.com/square/ghostunnel for production use. Maybe it's a better option than installing nginx, @ganto what do you think?
  • Write documentation.

@ganto With this roles you can customize of startup arguments for the exporters. For example:

prometheus_exporter__args:

  - name: 'node'
    options:
      - collector.textfile.directory: '/var/lib/prometheus/node-exporter'

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

@pedroluislopez
Copy link
Member Author

pedroluislopez commented Apr 15, 2020

Inventory examples.

Prometheus server:

prometheus_server__scrape_configs_configuration:

  - job_name: 'nginx'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/nginx.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'phpfpm'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/phpfpm.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'mysqld'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/mysqld.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'redis'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/redis.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'mongodb'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/mongodb.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'rabbitmq'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/rabbitmq.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

All nodes (by default it installs and configure node exporter):

prometheus_exporter__server: 'prometheus.domain.com
prometheus_exporter__allow: [ '<IP prometheus server>' ]

Nginx and PHP-FPM:

prometheus_exporter__append_groups: [ 'www-data' ]

prometheus_exporter__group_exporters: [ 'nginx' ]

prometheus_exporter__group_release_exporters:

  - name: 'phpfpm'
    resource: 'https://github.com/Lusitaniae/phpfpm_exporter/releases/download/v0.5.0/phpfpm_exporter-0.5.0.linux-amd64.tar.gz'
    archive: True
    binary: 'phpfpm_exporter-0.5.0.linux-amd64/phpfpm_exporter'

prometheus_exporter__group_ports_map:

  phpfpm: '9253'

prometheus_exporter__args:

  - name: 'nginx'
    options:
      - nginx.scrape-uri: 'https://{{ ca_migrations_fqdn }}/nginx_status'

  - name: 'phpfpm'
    options:
      - phpfpm.socket-paths:
           - '/run/php5.6-fpm-migrations.sock'
      - phpfpm.status-path: '/status.php'

MariaDB

prometheus_exporter__group_exporters: [ 'mysqld' ]

Redis

prometheus_exporter__group_release_exporters:

  - name: 'redis'
    resource: 'https://github.com/oliver006/redis_exporter/releases/download/v1.5.3/redis_exporter-v1.5.3.linux-amd64.tar.gz'
    archive: True
    binary: 'redis_exporter-v1.5.3.linux-amd64/redis_exporter'

prometheus_exporter__group_ports_map:

  redis: '9121'

prometheus_exporter__args:

  - name: 'redis'
    options:
      - redis.password: '{{ lookup("password", secret + "/redis/clusters/" + ansible_domain + "/password") }}'

MongoDB

prometheus_exporter__group_exporters: [ 'mongodb' ]

prometheus_exporter__args:

  - name: 'mongodb'
    options:
      - mongodb.uri: '{{ """mongodb://monitor:" +
                         lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                         "@localhost:27017""" }}'

RabbitMQ 3.8

prometheus_exporter__pre_installed_exporters: [ 'rabbitmq' ]

prometheus_exporter__group_ports_map:

  rabbitmq: '15672'

RabbitMQ < 3.8

prometheus_exporter__group_release_exporters:

  - name: 'rabbitmq'
    resource: 'https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC6.1/rabbitmq_exporter-1.0.0-RC6.1.linux-amd64.tar.gz'
    archive: True
    binary: 'rabbitmq_exporter-1.0.0-RC6.1.linux-amd64/rabbitmq_exporter'

prometheus_exporter__group_ports_map:

  rabbitmq: '9419'

prometheus_exporter__env:

  - name: 'rabbitmq'
    options:
      - RABBIT_USER: 'sensu'
        RABBIT_PASSWORD: '{{ lookup("password", secret + "/rabbitmq_server/accounts/sensu/password") }}'
        PUBLISH_ADDR: '{{ "localhost" if prometheus_exporter__pki|bool else prometheus_exporter__bind }}'
        PUBLISH_PORT: '{{ "3419" if prometheus_exporter__pki|bool else "9419" }}'

EDIT: [prometheus_exporter] Install repo exporter releases instead of compile go
EDIT 2: [prometheus_server] [prometheus_exporter] Add SSL feature with authentication support

@ganto
Copy link
Contributor

ganto commented Apr 16, 2020

@ganto With this roles you can customize of startup arguments for the exporters. For example:

Neat, I missed that 👍

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

Hmn, do you have an example? Can't find a single include_role statement in the current master branch.

@pedroluislopez
Copy link
Member Author

@ganto With this roles you can customize of startup arguments for the exporters. For example:

Neat, I missed that +1

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

Hmn, do you have an example? Can't find a single include_role statement in the current master branch.

073419e

…le go

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…tication support

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…rver, localhost dosen't work with TLS

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez
Copy link
Member Author

Prometheus rules example:

prometheus_server__rules:

  - name: 'example'
    rules:
      - alert: PrometheusTargetMissing
        expr: up == 0
        for: 5m
        labels:
          severity: error
        annotations:
          summary: !unsafe "Prometheus target missing (instance {{ $labels.instance }})"
          description: !unsafe "A Prometheus target has disappeared. An exporter might be crashed.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez
Copy link
Member Author

Prometheus alertmanager inventory example:

prometheus_alertmanager__server: 'prometheus.domain.com'
prometheus_alertmanager__allow: [ '<IP prometheus server>' ]
prometheus_alertmanager__global_configuration:
  smtp_from: prometheus@domain.com'
  smtp_smarthost: '...'
  smtp_auth_username: '...'
  smtp_auth_password: '...'
prometheus_alertmanager__receivers_configuration:
  - name: 'default-receiver'
    email_configs:
      - to: 'op@domain.com'

@pedroluislopez
Copy link
Member Author

Any news on this. I really would like to have Prom working as a role

Now the roles are completed. Next days i will write documentation.

…ocumentation

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez
Copy link
Member Author

Refactoring: use golang role for install exporters.
New examples in documentation.

ypid added 6 commits June 14, 2020 17:29
Mechanical edit done by (debops-optimize):

```Shell
git ls-files -z "ansible/roles/prometheus_*" | xargs --null -I '{}' find './{}' -type f -print0 | \
    xargs --null perl -i -p0e '
        s~
            ## Use Ansible 2.8 default shortcut: True or False

            ## Line: .*True
            \(?True\n?

            ## Line: if (ansible_local|d() and ansible_local.pki|d() and
            \s+if\s*\((?<var_ref1>ansible_local)\s*\|\s*d(?:efault)?\(\)\s+and\s+(?<var_ref2>\k<var_ref1>\.\w+)\s*\|\s*d(?:efault)?\(\)\s+and\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.pki

            ## Line: ansible_local.pki.enabled|d()|bool)
            \s+(?<expression>\k<var_ref2>\.\w+\s*\|\s*d(?:efault)?\(\))\)\n?
            ## expression: ansible_local.pki.enabled|d()|bool

            ## Line: else False
            \s+else\s+False\)?(?:\s*\|\s*bool)?
         ~$+{expression}~xg;

        s~
            ## Use Ansible 2.8 default shortcut: Thing if defined else fallback

            ## Line: .*ansible_local.core.domain
            \(?(?<var_ref3>(?<var_ref2>(?<var_ref1>ansible_local)\.\w+)\.\w+)\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.core
            ## var_ref3: ansible_local.core.domain

            ## Line: if (ansible_local|d() and ansible_local.core|d() and
            \s+if\s+\(\k<var_ref1>\s*\|\s*d(?:efault)?\(\)\s+and\s+\k<var_ref2>\s*\|\s*d(?:efault)?\(\)\s+and\n?

            ## Line: ansible_local.core.domain|d())
            \s+\k<var_ref3>(?:\s*\|\s*d(?:efault)?\(\))?\)\n?

            ## Line: else ansible_domain
            \s+else\s+\(?(?<else_exp>[^\n]+?)\)?\s+(?<end_marker>}}|%})
            ## else_exp: ansible_domain
        ~$+{var_ref3}|d($+{else_exp}) $+{end_marker}~xg;

        s~
            ## Use Ansible 2.8 default shortcut: More generic catch-all as last resort.

            ## Line: ansible_local|d() and ansible_local.pki|d() and
            (?<var_ref1>ansible_local)\s*\|\s*d(?:efault)?\(\)\s+and\s+(?<var_ref2>\k<var_ref1>\.\w+)\s*\|\s*d(?:efault)?\(\)\s+and\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.pki

            ## Line: ansible_local.pki.enabled|d()|bool
            \s+(?<expression>\k<var_ref2>\.\w+\s*\|\s*d(?:efault)?\(\))
            ## expression: ansible_local.pki.enabled|d()
         ~$+{expression}~xg;
    '
```
@ypid
Copy link
Member

ypid commented Jun 14, 2020

LGTM, thanks. I made a few improvements in Neuromobile#1. Note that I only tested the prometheus_server role (currently all I need).

pedroluislopez and others added 2 commits June 15, 2020 07:30
A few improvements for the Prometheus roles.
…depend server

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@julien-lecomte
Copy link
Contributor

Prometheus provides many different exporters other than the 'node' one (blackbox-exporter, etc...)
Shouldn't prometheus_exporter be better name with the full name prometheus_node_exporter ?

@pedroluislopez
Copy link
Member Author

pedroluislopez commented Sep 7, 2020

Prometheus provides many different exporters other than the 'node' one (blackbox-exporter, etc...)
Shouldn't prometheus_exporter be better name with the full name prometheus_node_exporter ?

The prometheus_exporter role allow install different exporters, and install node exporter by default. You can see a lot of examples in documentation.

@julien-lecomte
Copy link
Contributor

Any news on this?

author: 'Pedro Luis López'
description: 'Install and manage Prometheus alertmanager'
company: 'DebOps'
license: 'GPL-3.0-only'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But marked "GPL-3.0-or-later" in the header.

@pedroluislopez
Copy link
Member Author

Any news on this?

I have to do refactoring. @drybjed, we need first finish/approve Ghostunnel PR #1600, then I have to do refactoring in this PR to use the Ghostunnel role.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@hungpr0
Copy link
Contributor

hungpr0 commented Jul 27, 2021

Thanks a lot for the role.
Just want to ask if you can add task name on 2 import_role. ansible-lint is complaining. They should be:

- name: Import custom Ansible plugins
  import_role:
    name: 'ansible_plugins'

- name: Import DebOps secret role
  import_role:
    name: 'secret'

@hungpr0
Copy link
Contributor

hungpr0 commented Aug 5, 2021

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@pedroluislopez
Copy link
Member Author

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@hungpr0
Copy link
Contributor

hungpr0 commented Aug 5, 2021

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

https://github.com/Lusitaniae/apache_exporter
Apache-exporter uses

  • --telemetry.address instead of --web.listen-address
  • --telemetry.endpoint instead of --web.telemetry-path

@pedroluislopez
Copy link
Member Author

pedroluislopez commented Aug 8, 2021

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

https://github.com/Lusitaniae/apache_exporter
Apache-exporter uses

  • --telemetry.address instead of --web.listen-address
  • --telemetry.endpoint instead of --web.telemetry-path

@hungpr0

In this case, you have to drop common args and set args by each exporter. Like this:

prometheus_exporter__args:

  - name: 'common'
    state: 'absent'

  - name: 'node'
    options:
      - web.listen-address: '{{ ("localhost" if prometheus_exporter__pki|bool else prometheus_exporter__bind) + ":" +
                                (item.private_port if prometheus_exporter__pki|bool else
                                item.public_port) }}'

  - name: 'apache'
    options:
      - telemetry.address: '...'
      - telemetry.endpoint: '...'

@drybjed drybjed added this to the DebOps v3.2.0 milestone Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature requests to add new functionality priority: medium tag: monitoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants