[Question] can't specify blackbox as target

[shibumi]

Hello,
Do you have any idea why your ansible role is not fine with this config?

---
prometheus_targets:
  node:
    - targets:
      - 192.168.33.10:9100
      - 192.168.33.11:9100
  alertmanager:
    - targets:
      - "{{ inventory_hostname }}:9093"
  blackbox:
    - targets:
      - "{{ inventory_hostname }}:9115"

prometheus_scrape_configs:
- job_name: "prometheus"
  metrics_path: "/metrics"
  static_configs:
  - targets:
    - "{{ inventory_hostname }}:9090"
- job_name: "node"
  file_sd_configs:
  - files:
    - "/etc/prometheus/file_sd/node.yml"
- job_name: "alertmanager"
  file_sd_configs:
  - files:
    - "/etc/prometheus/file_sd/alertmanager.yml"
- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - "https://nullday.de"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: "{{ inventory_hostname }}:9115"

alertmanager_receivers:
- name: 'rocketchat'
  webhook_configs:
  - send_resolved: false
    url: 'https://chat.rz.tu-clausthal.de/hooks/sCyvZuLWYXG7erK6J/QsE2zGrpwfXrAjMgWD7KQKte3eY4mLAWFW6qKS92hFQ57RoJ'
alertmanager_route:
  group_interval: 1m
  repeat_interval: 1m
  receiver: 'rocketchat'

grafana_security:
  admin_user: admin
  admin_password: admin

grafana_datasources:
  - name: "Prometheus"
    type: "prometheus"
    access: "proxy"
    url: "http://{{ inventory_hostname }}:9090"
    isDefault: true
grafana_dashboards:
  - dashboard_id: '1860'
    revision_id: '8'
    datasource: '{{ grafana_datasources.0.name }}'
  - dashboard_id: '3662'
    revision_id: '2'
    datasource: '{{ grafana_datasources.0.name }}'
  - dashboard_id: '4271'
    revision_id: '3'
    datasource: '{{ grafana_datasources.0.name }}'

The error message I get is this one:

TASK [cloudalchemy.prometheus : Fail when file_sd targets are not defined in scrape_configs] **********************************************************************************************************************************************************************************************************************************
skipping: [192.168.33.10] => (item={'key': 'node', 'value': [{'targets': ['192.168.33.10:9100', '192.168.33.11:9100']}]}) 
skipping: [192.168.33.10] => (item={'key': 'alertmanager', 'value': [{'targets': ['192.168.33.10:9093']}]}) 
failed: [192.168.33.10] (item={'key': 'blackbox', 'value': [{'targets': ['192.168.33.10:9115']}]}) => {"changed": false, "item": {"key": "blackbox", "value": [{"targets": ["192.168.33.10:9115"]}]}, "msg": "Oh, snap! `blackbox` couldn't be found in you scrape configs. Please ensure you provided all targets from prometheus_targets in prometheus_scrape_configs"}

I am providing the blackbox target in the scrape_config, or not? So what is the issue here?

Another Problem I have is this:

Get http://192.168.33.10:9115/probe?module=http_2xx&target=https%3A%2F%2Fnullday.de: dial tcp 192.168.33.10:9115: connect: connection refused

Somehow the blackbox exporter doesn't get configured and started on my prometheus node. Did I miss something?

[paulfantom]

- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - "https://nullday.de"

It is a configuration for static_configs not for file_sd and prometheus_targets dict is ONLY for file_sd.

Look how you have job_name: "alertmanager" configuread and apply it to "blackbox".

[paulfantom]

As for the second issue, firewall?

Could you log into machine and run systemctl status blackbox_exporter?

[shibumi]

Thanks for the feedback.
I have tried a second run now with fresh images and this configuration:

---
prometheus_targets:
  node:
    - targets:
      - 192.168.33.10:9100
      - 192.168.33.11:9100
  alertmanager:
    - targets:
      - "{{ inventory_hostname }}:9093"

prometheus_scrape_configs:
- job_name: "prometheus"
  metrics_path: "/metrics"
  static_configs:
  - targets:
    - "{{ inventory_hostname }}:9090"
- job_name: "node"
  file_sd_configs:
  - files:
    - "/etc/prometheus/file_sd/node.yml"
- job_name: "alertmanager"
  file_sd_configs:
  - files:
    - "/etc/prometheus/file_sd/alertmanager.yml"
- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - "https://nullday.de"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: "192.168.33.11:9115"

alertmanager_receivers:
- name: 'rocketchat'
  webhook_configs:
  - send_resolved: false
    url: 'https://chat.rz.tu-clausthal.de/hooks/sCyvZuLWYXG7erK6J/QsE2zGrpwfXrAjMgWD7KQKte3eY4mLAWFW6qKS92hFQ57RoJ'
alertmanager_route:
  group_interval: 1m
  repeat_interval: 1m
  receiver: 'rocketchat'

grafana_security:
  admin_user: admin
  admin_password: admin

grafana_datasources:
  - name: "Prometheus"
    type: "prometheus"
    access: "proxy"
    url: "http://{{ inventory_hostname }}:9090"
    isDefault: true
grafana_dashboards:
  - dashboard_id: '1860'
    revision_id: '8'
    datasource: '{{ grafana_datasources.0.name }}'
  - dashboard_id: '3662'
    revision_id: '2'
    datasource: '{{ grafana_datasources.0.name }}'
  - dashboard_id: '4271'
    revision_id: '3'
    datasource: '{{ grafana_datasources.0.name }}'

This works as it should be. Thanks for your help. The special part about it is that I want blackbox exporters not on the prometheus. I want them on the clients.

[shibumi]

Btw it would be nice to have things like this:

It is a configuration for static_configs not for file_sd and prometheus_targets dict is ONLY for file_sd.

documented somewhere :)

[paulfantom]

Feel free to add to readme 😄

I thought this was enough: https://github.com/cloudalchemy/ansible-prometheus#relation-between-prometheus_scrape_configs-and-prometheus_targets but it seems that it isn't.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.