Send arbitrary extra headers when scraping

[jakedt]

It is currently possible to set the Authorization header when scraping hosts. I would like to extend this functionality to allow any arbitrary headers to be specified and set via config. Our particular use case is to send a signal to some proxy middleware, but there could definitely be other use cases sending information out of band directly to the hosts being scraped.

I think the implementation should be pretty simple, and I would take a stab at it if it is likely to get accepted. Thoughts?

[juliusv]

FWIW, someone is currently adding header support to the blackbox exporter, so there is some precedent: prometheus/blackbox_exporter#32

@brian-brazil what do you think about having this in Prometheus as well?

[brian-brazil]

In the case of the blackbox exporter, it's intended to be able to do arbitrary http blackbox probing so what makes sense there (which is basically all HTTP settings) doesn't necessarily make sense in Prometheus.

The question here is how complex do we want to allow scraping protocol to be, and how complex a knot are we willing to let users tie themselves in via the core configuration? Are we okay with making it easy for a scrape not to be quickly testable via a browser? At some point we have to tell users to use a proxy server to handle the more obscure use cases, rather than drawing their complexity into Prometheus.

As far as I'm aware the use case here relates to a custom auth solution with a non-recommended network setup. It's not unlikely that the next request in this vein would be to make these relabelable, and as this is an auth-related request, per discussion on #1176 we're not going to do that. I think we'd need a stronger use case to justify adding this complexity.

[juliusv]

Sounds reasonable.
On Jun 11, 2016 21:11, "Brian Brazil" notifications@github.com wrote:

In the case of the blackbox exporter, it's intended to be able to do
arbitrary http blackbox probing so what makes sense there (which is
basically all HTTP settings) doesn't necessarily make sense in Prometheus.

The question here is how complex do we want to allow scraping protocol to
be, and how complex a knot are we willing to let users tie themselves in
via the core configuration? Are we okay with making it easy for a scrape
not to be quickly testable via a browser? At some point we have to tell
users to use a proxy server to handle the more obscure use cases, rather
than drawing their complexity into Prometheus.

As far as I'm aware the use case here relates to a custom auth solution
with a non-recommended network setup. It's not unlikely that the next
request in this vein would be to make these relabelable, and as this is an
auth-related request, per discussion on #1176
#1176 we're not going to
do that. I think we'd need a stronger use case to justify adding this
complexity.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1724 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAg1mGts1EinMTeEaDWaSsqiBAOPrBbYks5qKwhsgaJpZM4IzJx_
.

[fabxc]

FWIW, JWT is not a custom auth and "recommended" is a very subjective term. I expect users to want their /metrics endpoints to integrate with their usual authentication setup. That means that not anyone can hit /metrics via the browser.

[brian-brazil]

JWT only requires the Authorization header, which we support. Asking for headers on top of that indicates something custom.

[juliusv]

Closing this since it seems like we're deciding against implementing this for now. Please reopen if I'm wrong.

[sra]

Just in case others come looking here for how to do this (there are at least 2 other issues on it), I've got a little nginx config that works. I'm not an nginx expert so don't mock! ;)

I run it in docker. A forward proxy config file for nginx listening on 9191:

http {
  map $request $targetport {
    ~^GET\ http://.*:([^/]*)/ "$1";
  }
  server {
    listen 0.0.0.0:9191;
    location / {
      proxy_redirect off;
      proxy_set_header NEW-HEADER-HERE "VALUE";
      proxy_pass  $scheme://$host:$targetport$request_uri;
    }
  }
}
events {
}

Run the transparent forward proxy:

docker run -d --name=nginx --net=host -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro nginx

In your prometheus job (or global) add the proxy_url key

  - job_name: 'somejob'
    metrics_path: '/something/here'
    proxy_url: 'http://proxyip:9191'
    scheme: 'http'
    static_configs:
    - targets:
      - '10.1.3.31:2004'
      - '10.1.3.31:2005'

[brian-brazil]

Are you sure that's the right config? I'm not an nginx expert but it seems to be taking part of the URL path and changing it to the port, rather than setting a header.

[sra]

It's working great! That part is needed to make it proxy to host:port pairs that are not known in advance. Well it is at least is the only way my limited nginx config experience found to do it.

[thangbn]

+1
For my particular case. We are using Cloud Foundry and can not access individual application instance.
CF support X-CF-APP-INSTANCE header to explicitly instruct cloud router to route request to a particular instance.
That's an ad-hoc case but anyway customizable is always better.

[robachmann]

@thangbn we are using Cloud Foundry too and setting custom headers in Prometheus would be most appreciated. I register applications in Consul and set X-CF-APP-INSTANCE as a tag which in a further step should be read and used by Prometheus in order to target individual instances. Until then, we use Prometheus' PushGateway to push instance specific metrics like memory-usage. I know PushGateway isn't intended for that use-case (https://prometheus.io/docs/practices/pushing/) but as long as we can't set headers for scraping we're bound to that work-around.

[eaglerainbow]

@thangbn @robachmann As you mention Cloud Foundry in this regard - for some days, there is a new kid on the block (Promregator), which is handling exactly your use case: scraping metrics from Clound Foundry applications using the X-CF-APP-INSTANCE approach. As said it's still quite fresh meat, but I would appreciate if you gave it a try providing your feedback.

[fitzoh]

Also came here looking for a way to set the X-CF-APP-INSTANCE header.

@juliusv any chance of reconsidering this one?

Cloud foundry has increased in popularity significantly since the issue was initially closed, and that particular use case has brought multiple people to this thread.

[brian-brazil]

If you're on Cloud Foundry you should look at something along the lines of https://github.com/promregator/promregator

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.