[Prometheus] Configure --web.external-url --web.route-prefix= and having a problem: Error loading available metrics!

[fchiorascu]

Proposal

Use case. Why is this important?
Because is something very used and there is a need of having secure under nginx.

Bug Report

Tested all possible options from internet, the only that not gives a 404 is the below one till now.

What did you expect to see?
I'd expect to have the web interface GUI for prometheus and to be able to see the metrics, right now i have an message: Error loading available metrics!

What did you see instead? Under which circumstances?
I've tried all the possible choices:
https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
https://www.robustperception.io/external-urls-and-path-prefixes
https://prometheus.io/docs/guides/tls-encryption/#nginx-example

Environment

• System information:
  Linux 3.10.0-862.14.4.el7.x86_64 x86_64

• Prometheus version:
  2.5.0

• Alertmanager version:
  0.15.3

• Prometheus configuration file:

# cat /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file /opt/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus --web.console.templates=/opt/prometheus/consoles --web.external-url=http://localhost:9090/prometheus --web.route-prefix=/prometheus --web.console.libraries=/opt/prometheus/console_libraries --storage.tsdb.min-block-duration=1h --storage.tsdb.max-block-duration=12h --storage.tsdb.retention=32d --log.level=debug

[Install]
WantedBy=default.target

• Alertmanager configuration file:

# cat /usr/lib/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager --config.file=/opt/alertmanager/alertmanager.yml --log.level=debug --web.external-url=http://localhost:9093/alertmanager --web.route-prefix=/alertmanager 

[Install]
WantedBy=default.target

• Logs:

From nginx server giving a curl:
[root@nginx1 ~]# curl 192.168.1.2:9090/prometheus
<a href="/prometheus/">Moved Permanently</a>.

[root@nginx1 ~]# curl 192.168.1.2:9090/prometheus/prometheus/
<a href="/prometheus/graph">Found</a>.


*Nginx config:
location /prometheus/ {
                auth_basic                    "[Monitoring] Prometheus";
                auth_basic_user_file /etc/nginx/.htpasswd;
                auth_request /_auth;
                proxy_set_header Host $host;
                proxy_pass http://prometheus_backend;
            }

upstream prometheus_backend {
server 192.168.1.2:9090;
}

[simonpasquier]

@lucperkins has done a tremendous job at documenting this part and there are even ready-to-go examples. In your case, I suspect that --web.route-prefix should be / instead of /prometheus.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[daenney]

You shouldn't use --web.route-prefix if --web.external-url already includes a path:

	a.Flag("web.external-url",
		"The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically.").
		PlaceHolder("<URL>").StringVar(&cfg.prometheusURL)

	a.Flag("web.route-prefix",
		"Prefix for the internal routes of web endpoints. Defaults to path of --web.external-url.").
PlaceHolder("<path>").StringVar(&cfg.web.RoutePrefix)

So either use a --web.external-url without /prometheus or drop the --web.router-prefix. Using both results in URLs like /prometheus/prometheus and you'll need to adapt your proxy config to account for that.

[fchiorascu]

Ok, thank you, i'll make a try.

[fchiorascu]

Doesn't make the things to work, still the same error and the I think I miss something.
What I need too correct here in prometheus.yml?

Prometheus.yml:

$ cat /opt/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s

alerting:
alertmanagers:

• static_configs:
  • targets:
    • localhost:9093

rule_files:

• all.rules

scrape_configs:

• job_name: prometheus
  static_configs:
  • targets: ['localhost:9090','localhost:9100']
    labels:
    env: prod
    alias: prometheus
• job_name: grafana
  scheme: https
  tls_config:
  insecure_skip_verify: true
  static_configs:
  • targets: ['localhost:3000']
    labels:
    env: prod
    alias: grafana
• job_name: alertmanager
  static_configs:
  • targets: ['localhost:9093']
    labels:
    env: prod

https://groups.google.com/forum/#!topic/prometheus-users/bA6rFgOLGTE

[petarkozic]

This works for me. I have HA setup with two alertmanager also behind Nginx with acme Let’s encrypt

[Service]
User=root
ExecStart=/opt/prometheus/prometheus \
    --config.file /opt/prometheus/prometheus.yml \
    --storage.tsdb.retention 100d \
    --storage.tsdb.path data/ \
    --web.external-url=https://localhost:9090/prometheus/ \
    --web.route-prefix=“/"

Nginx:

server {
       listen 80;
       server_name example.com;
       root _;

       location /.well-known/acme-challenge {
              root /var/www/letsencrypt;
       }
}

server {
        listen 443 ssl;
        server_name example.com;
        ssl_certificate     path to your cert;
        ssl_certificate_key path to your key;
        root _;

        location /prometheus {
        proxy_pass http://localhost:9090/prometheus/;

        proxy_set_header Accept-Encoding "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;


rewrite ^/prometheus/?$ /prometheus/graph redirect;
        rewrite ^/prometheus/(.*)$ /$1 break;

       
        }

}

And ywhen access, you access with:

https://example.com/prometheus

[fchiorascu]

Not working for me, Thank you.

$ curl http://localhost:9090/
<a href="/&#34;/&#34;">Found</a>.

[petarkozic]

You don't have anything on
http://localhost:9090/
if you setup web-url as I sent you.

Web external is:
--web.external-url=https://localhost:9090/prometheus/

[fchiorascu]

Prometheus:
$ cat /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file /opt/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus --web.console.templates=/opt/prometheus/consoles --web.console.libraries=/opt/prometheus/console_libraries --storage.tsdb.min-block-duration=1h --storage.tsdb.max-block-duration=12h --storage.tsdb.retention=32d --log.level=debug --web.external-url=https://localhost:9090/prometheus --web.route-prefix="/"

[Install]
WantedBy=default.target

CLI:

# curl localhost:9090/
<a href="/&#34;/&#34;">Found</a>.
# curl localhost:9090/metrics
404 page not found
# curl localhost:9090/prometheus/
404 page not found
# curl localhost:9090/prometheus/metrics
404 page not found

Nginx:

            location /prometheus {
                auth_basic                    "[Monitoring] Prometheus";
                auth_basic_user_file /etc/nginx/.htpasswd;
                rewrite ^/prometheus/?$ /prometheus/graph redirect;
                rewrite ^/prometheus/(.*)$ /$1 break;
                proxy_set_header Host $host;
                proxy_set_header Accept-Encoding "";
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://prometheus_backend/prometheus/;
            }

    upstream prometheus_backend {
        server 192.168.2.1:9090;
    }

[fchiorascu]

What i need to change in prometheus.yml?

[simonpasquier]

Nothing has to be changed in prometheus.yml. Try with --web.route-prefix=/ (unquoted).

[fchiorascu]

Yes it seems that behavior is strange changed the --web.route-prefix=/ (unquoted):

$  curl localhost:9090/
<a href="/prometheus/graph">Found</a>.

$ curl -s localhost:9090/metrics | head -10
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000104123
go_gc_duration_seconds{quantile="0.25"} 0.000185408
go_gc_duration_seconds{quantile="0.5"} 0.000219118
go_gc_duration_seconds{quantile="0.75"} 0.000279578
go_gc_duration_seconds{quantile="1"} 0.000854398
go_gc_duration_seconds_sum 0.011479145

[fchiorascu]

But still the same thing, this was the reason for asking about prometheus.yml:

[fchiorascu]

I've tried all the proposed configurations one by one till now...
Nothing is working when is about doing the query in GUI.

[fchiorascu]

Going on Nginx server and doing a query to reach Prometheus server the output is ok:
There is a real problem in nginx side.

# curl '192.168.20.6:9090/api/v1/query?query=up&time=2018-11-15T20:10:51.781Z'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","alias":"alertmanager","env":"prod","instance":"localhost:9093","job":"alertmanager"},"value":[1542312651.781,"1"]},{"metric":{"__name__":"up","alias":"grafana","env":"prod","instance":"localhost:3000","job":"grafana"},"value":[1542312651.781,"1"]},{"metric":{"__name__":"up","alias":

[fchiorascu]

-web.external-url=https://<proxyhost>/prometheus
the proxyhost must be the nginx (dns) configured not localhost from prometheus server?

[fchiorascu]

Someone has any idea about the best configuration/ simple that must work for prometheus.service and nginx?

[fchiorascu]

I have changed the /data directory 2 days ago as path from /data2 stopped prometheus and moved. Could be this a reason?