/metrics does not respect --web.external-url configuration

[gajus]

Bug Report

What did you do?

I have configured --web.external-url=http://gajus.com/prometheus/.

What did you expect to see?

All Prometheus service endpoints to be prefixed with a path that is part of the --web.external-url.

What did you see instead? Under which circumstances?

The host is respected, but the path is not respected, i.e. Prometheus attempts to look for http://gajus.com/metrics instead of http://gajus.com/prometheus/metrics.

Environment

• Prometheus version: v2.3.2
• Prometheus configuration file:

https://github.com/helm/charts/tree/master/stable/prometheus +

alertmanager:
  persistentVolume:
    size: '32Gi'
server:
  baseURL: https://gajus.com/prometheus/
  prefixURL: /prometheus
storage:
  local:
    retention: '720h'

[gajus]

I have raised an issue with Helm charts too, just in case this is a configuration error.

helm/charts#7088

[brian-brazil]

Can you share your configuration file?

This setting configures the Prometheus http server, it has no effect on the scrape logic.

[gajus]

Can you share your configuration file?

Using the default configuration (https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml) extended with the earlier mentioned configuration.

This setting configures the Prometheus http server, it has no effect on the scrape logic.

Point is that the http://gajus.com/prometheus/metrics endpoint exists and http://gajus.com/metrics does not. Since I have configured baseURL / prefixURL, I expect the checker to use the correct endpoint.

By the way, Prometheus does actually work as expected. I think it is just the liveness checker that is not aware of the prefix.

[brian-brazil]

That sounds like an issue with the Helm chart, Prometheus is working as documented.

[gajus]

That sounds like an issue with the Helm chart, Prometheus is working as documented.

Which part of Prometheus is working as expected... ?

If by baseURL is https://gajus.com/prometheus/, then the above should query http://localhost:9090/prometheus/metrics or this assumption flawed?

[brian-brazil]

Prometheus has no notion of baseURL, that's a Helm thing.

[gajus]

Prometheus has no notion of baseURL, that's a Helm thing.

--web.external-url, sorry.

https://github.com/helm/charts/blob/333bdfb086f70f4e8c3fe6f1c239418d6ff1e653/stable/prometheus/templates/server-deployment.yaml#L90-L92

Doesn't change the question.

[brian-brazil]

Yes, Prometheus is working as expected. You told it to scrape /metrics so that's what it tried to do.

[gajus]

Yes, Prometheus is working as expected. You told it to scrape /metrics so that's what it tried to do.

Which part of configuration is that?

[hoffie]

Which part of configuration is that?

I am not familiar with helm/charts at all, but if I were to guess, this is the place where another metrics path would have to be configured:
https://github.com/helm/charts/blob/4728544e0556cac4e52c70691f60c4ba31e36231/stable/prometheus/values.yaml#L789

No metrics_path is configured there, which means Prometheus takes the default of /metrics. This works as long as --web.external-url is not set. But as soon as it is modified, metrics_path would have to be adjusted (or rather: set at all) as well.

Relevant docs:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E

So you are probably right that this is an issue, but Prometheus-wise there is little which can be done as it is a configuration issue. It has therefore be fixed in the configuration files, which seem to be managed by helm/charts in your case. Therefore, It might make sense to open an issue in the helm/charts repository (maybe referencing this one).

[gajus]

No metrics_path is configured there, which means Prometheus takes the default of /metrics. This works as long as --web.external-url is not set. But as soon as it is modified, metrics_path would have to be adjusted (or rather: set at all) as well.

Here we go. We are down to the core issue.

I am saying that it Prometheus issue that after setting --web.external-url, the metrics path is not adjusted. After all, this is the exact purpose of the --web.external-url, right?

So you are probably right that this is an issue, but Prometheus-wise there is little which can be done as it is a configuration issue. It has therefore be fixed in the configuration files, which seem to be managed by helm/charts in your case. Therefore, It might make sense to open an issue in the helm/charts repository (maybe referencing this one).

helm/charts#7088

I have already raised an issue. It is a simple change knowing the requirement. But it isn't clear having read the docs whats the reason these are two separate configurations.

[geekodour]

EDIT: This comment might not be any relevant now because my next comment solves what I suggested here in configuration level

@brian-brazil changing the template to this should fix the display thing

 <a href="{{.URL | globalURL}}">{{.URL.Scheme}}://{{.URL.Host}}/{{pathPrefix}}{{.URL.Path}}</a><br>

but the where is .URL and $value evaluated?

<td class="endpoint">
              <a href="{{.URL | globalURL}}">{{.URL.Scheme}}://{{.URL.Host}}{{.URL.Path}}</a><br>
              {{range $label, $values := .URL.Query }}
                {{range $i, $value := $values}}
                  <span class="label label-primary">{{$label}}="{{$value}}"</span>
                {{end}}
              {{end}}
</td>

https://github.com/prometheus/prometheus/blob/master/web/ui/templates/targets.html#L43

[geekodour]

I got it working after setting the correct metrics_path . Here are my configurations:

apache virtualhost

 <VirtualHost *:443>
                ServerName servername.org
                ProxyPreserveHost on
                ProxyPass /prom/ http://localhost:9090/prom/
                ProxyPassReverse /prom/ http://localhost:9090/prom/
                <Location /prom>
                    AuthType Basic
                    AuthName "Restricted Content"
                    AuthUserFile /etc/apache2/.htpasswd
                    Require valid-user
                </Location>
                SSLEngine on
                SSLCertificateFile    /etc/letsencrypt/live/servername.org/cert.pem
                SSLCertificateKeyFile /etc/letsencrypt/live/servername.org/privkey.pem
                SSLCertificateChainFile /etc/letsencrypt/live/servername.org/chain.pem
</VirtualHost>

prometheus.yml

global:
  scrape_interval:     10s
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets:
        - 'localhost:9090'
    metrics_path: '/prom/metrics/'

prometheus.service (systemd)

[Unit]                                                                                                                                                                                         
Description=Prometheus Server                                                                                                                                                                  
                                                                                                                                                                                               
[Service]                                                                                                                                                                                      
User=prometheus                                                                                                                                                                                
Group=prometheus                                                                                                                                                                               
Type=simple                                                                                                                                                                                    
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.external-url=https://servername.org/prom/
ExecReload=/bin/kill -HUP $MAINPID                                                                                                                                                             
KillMode=mixed                                                                                                                                                                                 
Restart=always                                                                                                                                                                                 
                                                                                                                                                                                               
[Install]                                                                                                                                                                                      
WantedBy=multi-user.target

Now I can get the prometheus explorer at https://servername.org/prom/ (the tailing slash is important here)