Problems on setting up prometheus outside kubernetes cluster

[tuner23]

Hi,

i am trying to connect prometheus to a kubernetes-cluster. Prometheus is running outside the k8-cluster..

I created a bearer token, which works well when running with curl:
curl -X GET -H "Authorization: Bearer ..." https://ose-master.mgmt.tb/api/v1/services?resourceVersion=0

When i start up prometheus i get the following error(s):
ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: User "system:anonymous" cannot list all endpoints in the cluster component="kube_client_runtime" source="kubernetes.go:75" ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: User "system:anonymous" cannot list all services in the cluster component="kube_client_runtime" source="kubernetes.go:75" ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: User "system:anonymous" cannot list all pods in the cluster component="kube_client_runtime" source="kubernetes.go:75"
I also tried to inclube the bearer token in kubernetes_sd_configs with no difference.
Is that a problem in connecting ose-kubernetes with prometheus, or am i doing something wrong (since the request works well with curl)

Environment

• System information:

Linux 3.10.0-514.21.2.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 1.7.1 (branch: master, revision: 3afb3ff)
build user: root@0aa1b7fc430d
build date: 20170612-11:44:05
go version: go1.8.3

• Prometheus configuration file:
  scrape_configs:

• job_name: 'prometheus'
  scrape_interval: 5s
  static_configs:
  • targets: ['localhost:9090']
• job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  • role: endpoints
    api_server: 'https://ose-master.mgmt.tb'
    scheme: https
    tls_config:
    server_name: ose-master.mgmt.tb.noris.de
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
  • source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https
• job_name: 'kubernetes-nodes'
  scheme: https
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  • role: node
    api_server: 'https://ose-master.mgmt.tb'
    relabel_configs:
  • action: labelmap
    regex: _meta_kubernetes_node_label(.+)
• job_name: 'kubernetes-service-endpoints'
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  kubernetes_sd_configs:
  • role: endpoints
    api_server: 'https://ose-master.mgmt.tb'
    relabel_configs:
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: scheme
    regex: (https?)
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: metrics_path
    regex: (.+)
  • source_labels: [address, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: address
    regex: (.+)(?::\d+);(\d+)
    replacement: $1:$2
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_username]
    action: replace
    target_label: basic_auth_username
    regex: (.+)
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_password]
    action: replace
    target_label: basic_auth_password
    regex: (.+)
  • action: labelmap
    regex: _meta_kubernetes_service_label(.+)
  • source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

• Logs:

ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: User "system:anonymous" cannot list all endpoints in the cluster  component="kube_client_runtime" source="kubernetes.go:75"
ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:180: Failed to list *v1.Service: User "system:anonymous" cannot list all services in the cluster  component="kube_client_runtime" source="kubernetes.go:75"
ERRO[0002] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:181: Failed to list *v1.Pod: User "system:anonymous" cannot list all pods in the cluster  component="kube_client_runtime" source="kubernetes.go:75"

[tuner23]

Sorry, it was just a misconfiguration combined with an trailing on the bearer token.
The configuration should work with sth like:

• job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  • role: endpoints
    api_server: 'https://ose-master.mgmt.tb'
    • tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      scheme: https
      tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ...

[chesterlai]

@tuner23
Hi, I also met the same problem with you.
I am doubt there is something wrong on my tocken file
could you advice how do you fix your problem?

[tuner23]

Hi,
In my case there was a trailing CR (which was not visible via vim). Something like
/usr/bin/perl -pi -e 'chomp if eof' /var/run/secrets/kubernetes.io/serviceaccount/ose.token
helped..

[chesterlai]

thanks for your responding
however, it is different with my situation
there is normal in my token file

But I think it exists a problem about my setting.
My prometheus can't load token correctly, and it returns the same error with you.

Could you share your prometheus configuration with me?

And my token file format is which I copy the information on k8s secret "token" column directly.
Do I need to encode or decode that information first?
I have tried connect with curl , and it worked.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.