Prometheus Kubernetes Api Server context deadline exceeded

[blinqqt]

Bug Report

What did you expect to see?

Kubernetes-apiservers UP in prometheus target overview

What did you see instead? Under which circumstances?

Environment

• System information:

  AKS

• Prometheus version:

  image: docker.io/prom/prometheus:v2.3.1

• Prometheus configuration file:

- job_name: kubernetes-apiservers
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - default
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: kubernetes;https
    replacement: $1
    action: keep'

• Logs:
  The logs are created after the last deployment

level=info ts=2019-04-18T13:31:46.421606492Z caller=kubernetes.go:188 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-04-18T13:31:46.422988998Z caller=kubernetes.go:188 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-04-18T13:31:46.424009102Z caller=main.go:500 msg="Server is ready to receive web requests.

I receive the error message context deadline exceeded in the prometheus target overview. I already changed the "insecure_skip_verify" from false to true, but without success.
Is there any common solution for that? I tried to curl / wget from the prometheus pod to the api server, without success (curl not installed on pod, wget returns no output).

Please let me know, if more informations are nessecary from my side for this issue.

Thank you all in advanced.

[simonpasquier]

The error typically means that the endpoint is slow to respond. Try increasing the scrape_timeout & scrape_interval values. You can also start Prometheus with --log.level=debug.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.