more complete configuration examples

[hoppedmalt]

I've wrestled with wrapping together nginx for ssl wrapping, pushprox-proxy and -client; there apparently are some landmines that I am dancing on, but as there is no audible feedback like in real life, I have hard time telling when and where I took wrong turn.

Testing with curl:
proxyhost -> proxyclient/proxyclienthost: metrics about pushprox-client
proxyclienthost -> proxyclient/proxyclienthost: metrics about pushprox-client
proxyclienthost -> node_exporter/proxyclienthost: actual metrics

On startup, the pushprox-client manages to register with pushprox-proxy, but the garbage collection cycle cleans up the registration after a while. No actual data seems to be transmitted from node_exporter anywhere.

It would be really useful to have complete working configuration example as a starting point. Also, I would expect loglevel "debug" being at least slightly more verbose than "info", preferrably being very verbose on what the software is currently doing, what is failing and why.

[hoppedmalt]

Current setup and symptoms:

• prometheus on host A
• nginx as a ssl reverse proxy (:9111) and pushprox-proxy (:9112) on host B
• pushprox-client (:9114) and node_exporter (:9100) on virtual host C

....
Query from A to B:
openssl s_client -connect B:9111 -cert /path/to/prometheus.cert -key /path/to/prometheus.key

GET http://C:9114/metrics HTTP/1.1
Host: C

..results in metrics data from the proxy. B doesn't scrape C and eventually garbage collection cleans out C's entry from clients list. Only keepalive traffic between B and C is seen while scrape attempts are performed from A to C via pushprox-proxy running on B.

....

• The pushprox-client on C registers on pushprox-proxy on B on startup.
• Queries to C's metrics port give out only "go" and "pushprox" results.
• No queries happen from C's pushprox-client towards C's node_exporter.
• On all hosts, C resolves with an address that should be correct for connection to establish.

[hoppedmalt]

After furious debugging on this many-moving-things solution, it seems to be boiling down to nginx chopping away part of the request, ie.

GET http://C:9114/metrics HTTP/1.1
became
GET /metrics HTTP/1.1

..which naturally didn't work in satisfactory manner at all. After sacrificing some working mans sweat to the search engines, this was found: https://stackoverflow.com/questions/5834025/how-to-preserve-request-url-with-nginx-proxy-pass/45074723#45074723

nginx:

server {
    # http2 seems to break scrapes (latest checked prometheus version 2.32.1)
    #listen x.y.z.w:9111 ssl http2;
    listen x.y.z.w:9111 ssl;
    server_name _;

    # ssl config; use https://ssl-config.mozilla.org/ to get something useful

    # ssl_verify_client on; # use this to verify connecting clients certificate - use this, it is dangerous to go without.

    location / {
        rewrite            ^(.*)$   "://$http_host$uri$is_args$args";
        rewrite            ^(.*)$   "http$uri$is_args$args" break;
        proxy_set_header   Host     $host;

        proxy_pass "http://x.y.z.w:9112";
    }
}

for prometheus:

- job name: 'proxied'
  proxy_url: 'https://x.y.z.w:9111'
  tls_config:
    ca_file: '/path/to/ca.crt'
    key_file: '/path/to/prometheus_cert.key'
    cert_file: '/path/to/prometheus_cert.crt'
  file_sd_configs:
    - files:
    - '/path/to/proxied/nodes.d/*.json'

[menvol3]

Hello @hoppedmalt
I'm facing mostly the same issue with PushProx, which is located behind the Nginx ingress controller that uses HTTPS.

After adding the annotation nginx.ingress.kubernetes.io/proxy-read-timeout: "3600", I was able to see registered clients on the PushProx. However, after some time, they disappeared, and it looks like that was done by the garbage collector as you said.

My main error is that I'm receiving a 404 when Prometheus is reaching PushProx.
Can you please provide a more detailed explanation of how did you solved such a problem?