"no token found" with scheme: https

[alexanderfeder]

What did you do?
Scraping the same target (node exporter v0.14) via http and https (on the same host, both http and https are served through an Apache 2.4.25 reverse proxy, valid cert).

There is no difference in the metrics output between http and https when fetching via curl (apart from the obvious fluctuations of the numerical values)
Currently basic auth is configured, but also without that I get the same results.

Following command shows that there is no difference:

$ diff <(curl http://prometheus:xxxxx@mytargethost.com:80/metrics -s -S | awk '{ print $1 }') <(curl https://prometheus:xxxxx@mytargethost.com:443/metrics -s -S | awk '{ print $1 }')
$

A snippet of the output without removing the values using awk:

$ diff <(curl http://prometheus:xxxxx@mytargethost.com:80/metrics -s -S) <(curl https://prometheus:xxxxx@mytargethost.com:443/metrics -s -S)
12c12
< go_goroutines 38
---
> go_goroutines 23
15c15
< go_memstats_alloc_bytes 1.628576e+06
---
> go_memstats_alloc_bytes 2.335496e+06
18c18
< go_memstats_alloc_bytes_total 7.751765008e+09
---
> go_memstats_alloc_bytes_total 7.752471928e+09
24c24
< go_memstats_frees_total 8.4234285e+07
---
> go_memstats_frees_total 8.4236563e+07
30c30
< go_memstats_heap_alloc_bytes 1.628576e+06
---
> go_memstats_heap_alloc_bytes 2.335496e+06

[... snip ...]

What did you expect to see?
That both targets work, alas scraping via https does not and results in a "no token found"

What did you see instead? Under which circumstances?
The https target is not scraped by prometheus.

Environment

• System information:

Linux 4.9.0-3-amd64 x86_64

• Prometheus version:

prometheus, version 2.0.0-beta.0 (branch: master, revision: 2b5d9159537cbd123219296121e05244e26c0940)
  build user:       root@fc24486243df
  build date:       20170712-12:21:13
  go version:       go1.8.3

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
rule_files:
    - "alert.rules"
scrape_configs:
  - job_name: 'https-www-node-exporter'
    scheme: https
    basic_auth:
        username: prometheus
        password: theSamePassword
    static_configs:
      - targets: ['mytargethost.com']

  - job_name: 'http-www-node-exporter'
    basic_auth:
        username: prometheus
        password: theSamePassword
    static_configs:
      - targets: ['mytargethost.com']

• Logs:

WARN[0004] append failed                                 err="no token found" source="scrape.go:626" target="{__address__="mytargethost.com:443", __metrics_path__="/metrics", __scheme__="https", instance="mytargethost.com:443", job="https-www-node-exporter"}"

Do you have any ideas? Thanks a lot!

[fabxc]

We tested this ourselves right now. Scraping an https target worked fine.
Have you tried running the curls from the Prometheus instance itself?

[fabxc]

Closing since this is most likely a connectivity issue from the Prometheus instance. Please reopen if you found out something indicating otherwise.

[oxie]

Guyz i have to ask you because i am breaking my head over and over,

doing prometheus sd discovery while runing as app on DCOS, with new dcos version 1.11 i can get the metrics on port 61091 without a problem, but seeing on each node is exposed a different marathon metrics port 5050 (masters) 5051(slaves), so when targeting them i get the no token problem, i assuemed it might need marathon dcos bearer token or marathon_sd_configs module, but seems that its not working at all.

Any idea how to pass http token when using dns_sd_configs?

Thanks you!

[msglueck]

Howdy,
I know this issue has been closed, but there is no sense creating a new one. I have (had) the same issue on Prometheus 2.2.1 For me it was a Confluence custom exporter that showed the same issue. Configured with https scheme (with or without a access token as param). Prometheus showed "no token found", but clicking on the scrape url showd perfectly fine metrics (even validated those with promtool).

The URL is behind an Apache 2.4 reverse proxy which does the SSL termination (Confluence is hosting the endpoint with a Tomcat). The Apache has a typical HTTPS setup (RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent] and on 443 vhost RequestHeader set X-Forwarded-Proto "https").

We noticed when temporarily removing the HTTPS redirect in Apache and use the HTTP scheme Prometheus scraped the target.

The solution we found in the end was keeping the scrape URL in Prometheus as HTTP and let the Apache do the redirect to HTTPS.

So to sum it up: there seems to be an issue with https scheme that sometimes pops up, but I wasn't able to get more info out of Prometheus on the cause. So if you can give some hints on how to debug this, I am happy to help out.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.