password_file ignored in kubernetes_sd_configs

[drewboswell]

Bug Report

What did you do?
Attempted to use password_file in stead of plaintext password in kubernetes_sd_configs

What did you expect to see?
Authorized http calls

What did you see instead? Under which circumstances?
Unauthorized/401 codes thrown in response

In kubernetes_sd_configs only:

• if I use password, authorized, it works
• if I use password_file, unauthorized, it doesn't work
• if I use password and password_file at the same time, no error thrown, but it works

it appears that the password_file variable is ignored completely

Environment

• System information:
  Linux 4.15.0-45-generic x86_64

• Prometheus version:
  prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f)
  build user: root@f9f82868fc43
  build date: 20190131-11:16:59
  go version: go1.11.5

• Prometheus configuration file:

# Global Prometheus Configuration
global:
  scrape_interval: 30s     # By default, scrape targets every 30 seconds.
  evaluation_interval: 30s # By default, scrape targets every 30 seconds.

scrape_configs:

- job_name: kubernetes-apiservers-pre
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://**********:443
    tls_config:
      insecure_skip_verify: true
    basic_auth:
      username: prometheus
      password_file: /var/run/secrets/prometheus/pre
  scheme: https
  tls_config:
    insecure_skip_verify: true
  basic_auth:
    username: prometheus
    password_file: /var/run/secrets/prometheus/pre
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

• Logs:

prometheus_1  | level=debug ts=2019-02-13T11:16:30.775409678Z caller=klog.go:70 component=k8s_client_runtime func=Infof msg="GET https://********:443/api/v1/endpoints?limit=500&resourceVersion=0 401 Unauthorized in 2 milliseconds"
prometheus_1  | level=error ts=2019-02-13T11:16:30.775560501Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: Unauthorized"

[simonpasquier]

Indeed... I've tested #5211 quickly and it seems to be ok but it would good if you could test it too.

[drewboswell]

Wow that was fast, yes I just checkout out your pull request built it and it works like a charm. It also throws the usual errors if the file doesn't exist etc.

Thanks a bunch!

[drewboswell]

There is a problem when using the "bearer_token_file" within the kubernetes SD block but it is a different error that I get. #5221