Proxy to allow Prometheus to scrape through NAT etc.
Switch branches/tags
Nothing to show
Clone or download
brian-brazil Merge pull request #36 from rollulus/metrics
Instrument client and proxy
Latest commit 0581e4a Nov 24, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Added CircleCI test Nov 14, 2017
client Instrument client with error counters Nov 19, 2018
docs Add sequence diagram to README (#30) Oct 31, 2018
proxy Instrument proxy http handlers Nov 19, 2018
util updated to kingpin logs Sep 18, 2017
vendor updated to kingpin logs Sep 18, 2017
LICENSE Move, add license Jul 25, 2017
README.md Add sequence diagram to README (#30) Oct 31, 2018

README.md

PushProx CircleCI

PushProx is a client and proxy that allows transversing of NAT and other similar network topologies by Prometheus, while still following the pull model.

While this is reasonably robust in practice, this is a work in progress.

Running

First build the proxy and client:

go get github.com/robustperception/pushprox/{client,proxy}
cd ${GOPATH-$HOME/go}/src/github.com/robustperception/pushprox/client
go build
cd ${GOPATH-$HOME/go}/src/github.com/robustperception/pushprox/proxy
go build

Run the proxy somewhere both Prometheus and the clients can get to:

./proxy

On every target machine run the client, pointing it at the proxy:

./client --proxy-url=http://proxy:8080/

In Prometheus, use the proxy as a proxy_url:

scrape_configs:
- job_name: node
  proxy_url: http://proxy:8080/
  static_configs:
    - targets: ['client:9100']  # Presuming the FQDN of the client is "client".

If the target must be scraped over SSL/TLS, add:

  params:
    _scheme: [https]

rather than the usual scheme: https. Only the default scheme: http works with the proxy, so this workaround is required.

Service Discovery

The /clients endpoint will return a list of all registered clients in the format used by file_sd_configs. You could use wget in a cronjob to put it somewhere file_sd_configs can read and then then relabel as needed.

How It Works

Sequence diagram

Clients perform scrapes in a network environment that's not directly accessible by Prometheus. The Proxy is accessible by both the Clients and Prometheus. Each client is identified by its fqdn.

For example, the following sequence is performed when Prometheus scrapes target fqdn-x via PushProx. First, a Client polls the Proxy for scrape requests, and includes its fqdn in the poll (1). The Proxy does not respond yet. Next, Prometheus tries to scrape the target with hostname fqdn-x via the Proxy (2). Using the fqdn received in (1), the Proxy now routes the scrape to the correct Client: the scrape request is in the response body of the poll (3). This scrape request is executed by the client (4), the response containing metrics (5) is posted to the Proxy (6). On its turn, the Proxy returns this to Prometheus (7) as a reponse to the initial scrape of (2).

Security

There is no authentication or authorisation included, a reverse proxy can be put in front though to add these.

Running the client allows those with access to the proxy or the client to access all network services on the machine hosting the client.