[question] Dynamic Prometheus configuration

[hgontijo]

I'd like to automate the process of adding more configuration after Prometheus installation.
My use case is to add more blackbox configuration (under scrape_configs) to prometheus.yml on-the-fly. Is there a way to setup multiple Prometheus config files?

Environment

prometheus, version 1.1.3 (branch: master, revision: ac374aa)
build user: root@3e392b8b8b44
build date: 20160916-11:36:30
go version: go1.6.3

[juliusv]

@hgontijo There's no support for multiple config files - if you want to change arbitrary things in the config while Prometheus is running, you can just send a HUP signal to Prometheus and it will reload the config.

If all you want to change are the targets, you can use any of the dynamic service discovery mechanisms, including file_sd_config, which allows you to let Prometheus watch the list of targets in a set of files.

[hgontijo]

Great, file_sd_configs worked nicely with my blackbox job.

As far the refresh_interval, the documentation says

# Refresh interval to re-read the files.
[ refresh_interval: <duration> | default = 5m ]

... however the refresh rate for me was a couple of secs.

I tried to configure the refresh_interval as:

scrape_configs:
  - job_name: 'blackbox-http-2xx-secure'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    file_sd_configs:
      - files:
        - '/etc/prometheus/blackbox/*.yml'
      - refresh_interval: 10s
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter.

And it gives this error message:

time="2016-09-30T19:23:31Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): file service discovery config must contain at least one path name" source="main.go:126"

it works fine without the refresh_interval on the job configuration.

[juliusv]

@hgontijo That is odd. Refreshes resulting from the refresh timer should execute the same code path as the initial load or refreshes triggered by changes to the files.

By the way: you probably do not need to set your refresh interval to 10s. The file_sd SD mechanism automatically detects changes to any of the watched target files, so the refresh interval is only a safety guard in case a watched update ever gets lost due to some bug or something. Then it will still eventually get the new target state. Normally you should be able to keep it at 5 minutes or so.

[hgontijo]

@juliusv the error message happens when I configure refresh_interval under file_sd_configs and restart (stop/start) the service. As you suggested, I'll go without refresh_interval since the file change detection works fine.

[nuaays]

@hgontijo you can refer here for config refresh_interval :-)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.