bearer_token_file option not working in kubernetes_sd_configs

[drewboswell]

Bug Report

What did you do?
Attempted to use bearer_token_file in stead of plaintext bearer_token in kubernetes_sd_configs

What did you expect to see?
Authorized http calls

What did you see instead? Under which circumstances?
net/http: invalid header field value "Bearer *******************\n" for key Authorization

In kubernetes_sd_configs only:

• if bearer_token_file set, loads data, but throws errors (see logs below). It looks like there is a newline character added even though the file does not contain one.

Environment

• System information:
  Linux 4.15.0-45-generic x86_64

• Prometheus version:
  prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f)
  build user: root@f9f82868fc43
  build date: 20190131-11:16:59
  go version: go1.11.5

• Prometheus configuration file:

# Global Prometheus Configuration
global:
  scrape_interval: 30s     # By default, scrape targets every 30 seconds.
  evaluation_interval: 30s # By default, scrape targets every 30 seconds.

scrape_configs:

- job_name: kubernetes-apiservers-pre
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://**********:443
    tls_config:
      insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/prometheus/pre
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/prometheus/pre
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

• Logs:

prometheus_1         | level=error ts=2019-02-14T16:40:10.193074594Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:306: Failed to list *v1.Endpoints: Get https://*********:443/api/v1/endpoints?limit=500&resourceVersion=0: net/http: invalid header field value \"Bearer *******************\\n\" for key Authorization"
prometheus_1         | level=debug ts=2019-02-14T16:40:10.19404676Z caller=klog.go:53 component=k8s_client_runtime func=Verbose.Infof msg="Listing and watching *v1.Service from github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:307"
prometheus_1         | level=debug ts=2019-02-14T16:40:10.194199661Z caller=klog.go:70 component=k8s_client_runtime func=Infof msg="GET https://*********:443/api/v1/services?limit=500&resourceVersion=0  in 0 milliseconds"

[simonpasquier]

I suspect that your bearer token file has a trailing \n but in any case it should have been removed. Hopefully it will get fixed by the latest version of #5211 too.

[drewboswell]

I suspect that your bearer token file has a trailing \n but in any case it should have been removed. Hopefully it will get fixed by the latest version of #5211 too.

Nope, there is no trailing \n no \r or combination of other evil special chars in the token file.

BTW I tested with the Pull-request version, and it still does not work. You should be able to reproduce this extremely easily. Just use any token in a token file and it should break.

[simonpasquier]

Ok thanks for the heads-up. I'll try this on monday!

[simonpasquier]

I've tried with the latest version of #5211 and the issue goes away. The current version in master doesn't trim leading and trailing white spaces.