Prometheus gets 503 when scraping target behind haproxy

[quolix]

What did you do?
What did you expect to see?
What did you see instead? Under which circumstances?

We run pushgateway on mesos, and put it behind haproxy. curl and chrome can fetch the URL http://pushgateway.corp.quobyte.com:80/metrics with no problems (that's the one displayed in /targets), but prometheus shows a "server returned HTTP status 503 Service Unavailable" in /targets.

I'll dig deeper when I find time, but I thought I'll document the issue for others.

Environment

• System information:

  Linux 3.10.0-327.18.2.el7.x86_64 x86_64

• Prometheus version:

  0.19.2 / 23ca13c

• Prometheus configuration file:

scrape_configs:
  - job_name: "pushgateway"
    target_groups:
      - targets:
        - pushgateway.corp.quobyte.com

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[fabxc]

Can you als curl the URL from the Prometheus node?

[matthiasr]

There is a trap in how Prometheus constructs tree host header and how HAProxy handles it. I suppose you are using ACLs for host-based routing? Could you try `curl -v -H 'Host: pushgateway.corp.quobyte.com:80 http://pushgateway.corp.quobyte.com/` (note the port in the host header)? The HTTP standard days they are equivalent, but because HAProxy doesn't actually know the semantics of the host header you need to reflect that in the ACL.

[quolix]

@matthiasr yes, this yields a 503. Adding an addtional URL with :80 in the Mesos label fixes it then:

 "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_VHOST": "pushgateway.corp.quobyte.com,pushgateway.corp.quobyte.com:80"
  },

Thank you very much for the hint! I hope this helps others if they run into the same issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.