Prometheus always uses last proxy_url in alertmanagers config

[hoffie]

What did you do?
When using proxy_url in the alertmanagers config section, prometheus does not behave as expected. It always seems to use the proxy configuration of the last entry instead of the per-entry value.

What did you see instead? Under which circumstances?

level=error ts=2017-12-29T21:26:46.585763037Z caller=notifier.go:444 component=notifier alertmanager=http://127.0.0.1:9000/api/v1/alerts count=1 msg="Error sending alert" err="Post http://127.0.0.1:9000/api/v1/alerts: proxyconnect tcp: dial tcp 127.0.0.1:9001: getsockopt: connection refused"
level=error ts=2017-12-29T21:26:46.586603048Z caller=notifier.go:444 component=notifier alertmanager=http://127.0.0.1:8000/api/v1/alerts count=1 msg="Error sending alert" err="Post http://127.0.0.1:8000/api/v1/alerts: proxyconnect tcp: dial tcp 127.0.0.1:9001: getsockopt: connection refused"

Note that proxyconnect both times tries accessing port 9001

What did you expect to see?
proxy_url should work per-alertmanager config according to docs and scrape_configs.
In above output, the alertmanager on port 8000 should be accessed by the proxy on port 8001 (not 9001).

Environment

• System information: Linux 4.14.8-1-ARCH x86_64 (also verified on RHEL7.4)
• Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

• Prometheus configuration file:

global:
  evaluation_interval: 5s
  scrape_interval: 5s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:8000
    proxy_url: http://127.0.0.1:8001
    path_prefix: /a

  - static_configs:
    - targets:
      - 127.0.0.1:9000
    proxy_url: http://127.0.0.1:9001
    path_prefix: /b

rule_files:
  - rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

• rules.yml configuration file:

groups:
- name: foo
  rules:
  - alert: foo
    expr: count(up) >= 0

[brian-brazil]

Can you try master? I believe this has already been fixed.

[hoffie]

@brian-brazil: Thanks for the quick reply. I already tried, but ran into another issue (#3637). Meanwhile, I found a8cce41 to be a compiling >2.0.0 version which indeed already fixes this issue. So, sorry for the noise.

The error message now states the proper port and I assume that in case of real proxies sitting there it would also work as expected:

level=error ts=2017-12-29T23:10:54.210402217Z caller=notifier.go:444 component=notifier alertmanager=http://127.0.0.1:9000/b/api/v1/alerts count=1 msg="Error sending alert" err="Post http://127.0.0.1:9000/b/api/v1/alerts: proxyconnect tcp: dial tcp 127.0.0.1:9001: getsockopt: connection refused"
level=error ts=2017-12-29T23:10:54.210619085Z caller=notifier.go:444 component=notifier alertmanager=http://127.0.0.1:8000/a/api/v1/alerts count=1 msg="Error sending alert" err="Post http://127.0.0.1:8000/a/api/v1/alerts: proxyconnect tcp: dial tcp 127.0.0.1:8001: getsockopt: connection refused"

Thanks for creating and maintaining the whole project/concept/ecosystem of Prometheus! :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.