file_sd_config: Support custom metrics_path

[klausenbusk]

Proposal

Use case. Why is this important?

We have around 300 Linux machine at different customers behind NAT, as they are behind NAT we use SSH to setup remote port forwarding, so Prometheus can scrape the node_exporter metrics from them. It works, but SSH doesn't scale very well (~ 1,5MB per machine) and security wise it isn't ideal (we need to trust the "machine" as OpenSSH doesn't provide any control over remote port forwarding (all or nothing)).

In the progress of evaluating other options, I did discover that Promethues doesn't provide a easy to configure multiple metrics_path for a target.

Right now we use a file_sd_config, like so:

  {
    "targets": [
      "ssh-company-tld.default:<port>"
    ],
    "labels": {
      "macaddress": "<id>"
    }
  },

and my idea was to built a custom SSH/proxy/http server on top of Golang ssh package. So I would end up with a SSH server the machine connect to and a HTTP server Prometheus scrape from.

Something like, but I can't do that, as Prometheus doens't provide a option to adjust the path per "target" (I'm not sure what the right term is).

GET /metrics/<id>

Ideally I could do something like:

  {
    "targets": [
      "ssh-company-tld.default:80/metrics/<id>"
    ],
    "labels": {
      "macaddress": "<id>"
    }
  },

[brian-brazil]

This is already possible.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[klausenbusk]

This is already possible.

How? The only way I can think of, is either listen on a "lot" of ports or creating a custom Prometheus config generator (which create a scrape_job for every single machine and set metrics_path).

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

So I should ask the community for workarounds instead of opening a feature request? How does that make sense? I'm wondering if we are talking past each other?

[klausenbusk]

This is already possible.

Sorry for the noise. I did spend some more time on this and did stumble on #4001, which works perfectly for my setup. I wasn't aware of that I could simply change __metrics_path__. I got the impression, that is wasn't possible after reading #1852, but it is a old issue so a lot has changed since.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.