prom not loading alerting rules

[m247support]

Hi,

I am playing around with Prometheus for the first time and for some reason I am not able to get Prometheus to load a test alerting.rules file. Can anyone see what I might be missing / doing wrong?

Running the official containers as follows:

docker run -d \
        --restart unless-stopped \
        -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml \
        -v $PWD/alerting.rules:/etc/prometheus/alerting.rules \
        -p 9090:9090
        --name prom \
        prom/prometheus:v1.5.0 -alertmanager.url=http://127.0.0.1:9093 -config.file=/etc/prometheus/prometheus.yml

docker run -d \
        -p 9093:9093 \
        -v $PWD/alertmanager.yml:/etc/alertmanager.yml \
        --name=alertmanager \
        prom/alertmanager:v0.5.1 -config.file=/etc/alertmanager.yml

Thanks in advance for any help!

Environment

$ docker -v
Docker version 1.11.2, build b9f10c9

• System information:

$ uname -srm
Linux 4.4.0-62-generic x86_64

• Prometheus version:

    /bin # ./prometheus --version
    prometheus, version 1.5.0 (branch: master, revision: d840f2c400629a846b210cf58d65b9fbae0f1d5c)
      build user:       root@a04ed5b536e3
      build date:       20170123-13:56:24
      go version:       go1.7.4
  

• Alertmanager version:

    /alertmanager # /bin/./alertmanager --version
    alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
      build user:       root@fb407787b8bf
      build date:       20161125-08:14:40
      go version:       go1.7.3
  

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

  external_labels:
      monitor: 'prom'

  rule_files:
    - '/etc/prometheus/alerting.rules'

scrape_configs:
  - job_name: dummy
    consul_sd_configs:
      - server: '172.17.0.1:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,internal,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

• Alertmanager configuration file:

global:
  smtp_smarthost: '172.17.0.1:25'
  smtp_from: 'alertmanager@somedomain'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'somepasswordhere'

route:
  group_by: ['service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-X-mails

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'somemailbox@somedomain'

ALERT TestAlert
  IF up == 1
  FOR 1s
  ANNOTATIONS {
    summary = "Instance {{ $labels.instance }} up",
    description = "{{ $labels.instance }} of job {{ $labels.job }} has been up for more than 5 minutes.",
  }

promtool check-rules alerting.rules                                                                                                                                                               
Checking alerting.rules
  SUCCESS: 1 rules found

• Logs:

$   docker logs -f prom
time="2017-02-07T04:02:04Z" level=info msg="Starting prometheus (version=1.5.0, branch=master, revision=d840f2c400629a846b210cf58d65b9fbae0f1d5c)" source="main.go:75"
time="2017-02-07T04:02:04Z" level=info msg="Build context (go=go1.7.4, user=root@a04ed5b536e3, date=20170123-13:56:24)" source="main.go:76"
time="2017-02-07T04:02:04Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"
time="2017-02-07T04:02:05Z" level=info msg="Loading series map and head chunks..." source="storage.go:373"
time="2017-02-07T04:02:05Z" level=info msg="0 series loaded." source="storage.go:378"
time="2017-02-07T04:02:05Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
time="2017-02-07T04:02:05Z" level=info msg="Listening on :9090" source="web.go:259"
time="2017-02-07T04:07:05Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:611"
time="2017-02-07T04:07:05Z" level=info msg="Done checkpointing in-memory metrics and chunks in 66.037575ms." source="persistence.go:638"
…

$   docker logs -f alertmanager 
time="2017-02-07T04:02:02Z" level=info msg="Starting alertmanager (version=0.5.1, branch=master, revision=0ea1cac51e6a620ec09d053f0484b97932b5c902)" source="main.go:101"
time="2017-02-07T04:02:02Z" level=info msg="Build context (go=go1.7.3, user=root@fb407787b8bf, date=20161125-08:14:40)" source="main.go:102"
time="2017-02-07T04:02:02Z" level=info msg="Loading configuration file" file="/etc/alertmanager.yml" source="main.go:195"
time="2017-02-07T04:02:03Z" level=info msg="Listening on :9093" source="main.go:250"
time="2017-02-07T04:17:02Z" level=info msg="running maintenance" component=silences source="silence.go:233"
time="2017-02-07T04:17:02Z" level=info msg="running maintenance" component=nflog source="nflog.go:288"
time="2017-02-07T04:17:02Z" level=info msg="maintenance done" component=silences duration=500.258µs source="silence.go:250"
time="2017-02-07T04:17:02Z" level=info msg="maintenance done" component=nflog duration=244.919µs source="nflog.go:305"
time="2017-02-07T04:32:02Z" level=info msg="running maintenance" component=silences source="silence.go:233"
time="2017-02-07T04:32:02Z" level=info msg="running maintenance" component=nflog source="nflog.go:288"
…

[brian-brazil]

Why do you think it isn't loaded?

[linsheng9731]

see this -> #1866

[brancz]

Be aware that mounting single files in docker does not remount a file when you modify it, when you mount a single file in docker it will never update. Therefore if you do want to run Prometheus in this way, it is recommended to mount your entire directory into /etc/prometheus, assuming there is nothing in your current working directory other than those config files.

[m247support]

Hi, thanks all for the help!

@brian-brazil I believe the alerting.rules file is not loaded because within Prometheus, the alert section states "No alerting rules defined" and nothing is listed under Status > Rules. Is my assumption correct?

@linsheng9731 Thanks for this tip! Unfortunately, it did not work in my case, but it will be very useful to know for future reference.

@brancz There are some sym link /etc/prometheus, including console_libraries -> /usr/share/prometheus/console_libraries, consoles -> /usr/share/prometheus/consoles. Are these automatically every time when prometheus starts up? Further, is it considered safe to mount a directory containing symlinks to the host fs? If not, I suppose I could mount another directory within /etc.

[m247support]

Hi,

I have tried to mount the directory containing prometheus.yml and alerting.rules files /etc/conf instead and configured prometheus appropriately, but prometheus only seems to load prometheus.yml (not the rules file). I have tried using the example rules at https://prometheus.io/docs/alerting/rules/ rather than the rule previously stated, without much effect. I have noticed logs sent to stdout seem to be info level only. Is the logging level of the official prometheus image set to info? If so, is there a way to configure level to debug without having to compile a new binary (i.e. any flags that can be set)? Any further thoughts welcome.

[m247support]

So it turns that indentation is really, really important for yaml (note that I had incorrectly nested rule_files under global).

Closing this off but thanks for your help!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.