Alertmanager PathPrefix not used by prometheus

[groob]

What did you do?

Used alertmanager config below in my prometheus.yml file

alerting:
  alertmanagers:
  - scheme: http
    path_prefix: "/alertmanager"
    static_configs:
    - targets:
        - "alertmanager.default.svc.cluster.local"

What did you expect to see?

A successful alert sent to the alertmanager.

What did you see instead? Under which circumstances?

time="2016-12-16T02:32:37Z" level=error msg="Error sending alerts: bad response status 404 Not Found" alertmanager="http://alertmanager.default.svc.cluster.local:80/api/v1/alerts" count=2 source="notifier.go:335"

• Prometheus version:

prometheus, version 1.4.1 (branch: master, revision: 2a89e8733f240d3cd57a6520b52c36ac4744ce12)
  build user:       root@e685d23d8809
  build date:       20161128-09:59:22
  go version:       go1.7.3* 

I'm able to send alerts using curl successfully

curl -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert1"}}]' alertmanager.default.svc.cluster.local/alertmanager/api/v1/alerts

If I remove the -web.externalurl flag from alertmanager, prometheus also succesfuly sends alerts.

[groob]

I tried debugging this myself. The path_prefix appears to load correctly. And alertmanagerFromGroup
https://github.com/prometheus/prometheus/blob/master/notifier/notifier.go#L477
also seems to correctly add the prefix from the config.

But I couldn't find out where Sync() is called from https://github.com/prometheus/prometheus/blob/master/notifier/notifier.go#L442

[brancz]

Sync() is part of the Syncer interface and called here. @fabxc is probably the one who will find the problem most quickly as he implemented the feature and is most familiar with the discovery implementation as well. I'll try to find it as well.

[groob]

Wow, sorry. this actually turned out to be user error.

My yaml looked like

alerting:
  alertmanagers:
  - path_prefix: "/alertmanager"
  - scheme: http
    static_configs:
    - targets:
        - "alertmanager.default.svc.cluster.local"

Then I fixed it, but didn't reload the config, and spent a good 2 hours debugging this.

On the plus side, I now understand how a good part of the notifier and config loading code works.

[nelsonfassis]

And how does it look now @groob ? I'm having the same issue as you.

` ```
alerting:
alertmanagers:
- scheme: http
path_prefix: "/alertmanager"
static_configs:
- targets:
- alertmanager-0.alertmanager:9093
- alertmanager-1.alertmanager:9093

`

[nelsonfassis]

Found the issue to be with path_prefix syntax.
It should be like:

• path_prefix: "alertmanager/"
  Changed the "/" to the end and it works for me now.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.