server returned HTTP status 401 Unauthorized

[ghost]

What did you do?
Run Prometheus on k8s

What did you expect to see?
Prometheus could tell me the endpoint status

What did you see instead? Under which circumstances?
Prometheus reported error "server returned HTTP status 401 Unauthorized"

Environment

• System information:
  Linux 3.10.0-514.21.2.el7.x86_64 x86_64

• Prometheus version:
  docker.io/prom/prometheus latest 4da113bb6ae3

• Alertmanager version:
  docker.io/prom/alertmanager latest 822973257511

• Prometheus configuration file:

   - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
        - role: endpoints

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

• Logs:

kubernetes-service-endpoints
--

http://172.32.0.64:9093/metrics | UP | app="alertmanager" instance="172.32.0.64:9093" kubernetes_name="alertmanager" | 11.684s ago |  
http://172.32.8.66:3000/api/metrics | DOWN | app="grafana" instance="172.32.8.66:3000" kubernetes_name="grafana" | 12.938s ago | server returned HTTP status 401 Unauthorized

[ghost]

when I use grafana api key to visit "http://172.32.1.9:3000/api/metrics" url, it will be ok. for example:
curl -H \"Authorization: Bearer eyJrIjoiOTN5U0dpNnFrTkg2VVlabVRJcjdnaXlrc0NkYzc0MlAiLCJuIjoidmlld2VyS2V5IiwiaWQiOjF9\" http://172.32.1.9:3000/api/metrics

So the problem is how to add a grafana api key in prometheus.yml ? @brian-brazil

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.