prometheus get kube-scheduler/kube-controller-manager metrics via https?

[szediktam]

Bug Report

What did you do?
I want to use prometheus-oprator to get metrics of kube-scheduler/kube-controller-manager via https.
I use certificate-authority-data, client-certificate-data and client-key-data in /etc/kubernetes/scheduler.conf to generate a secret. And use this secret to mount them into servicemonitor as caFile, certFile and keyFile.
And the endpoints is https://X.X.X.X:10259/metrics.

BUT, the kube-scheduler target generate a error:
Get https://X.X.X.X:10259/metrics: x509: certificate is valid for 127.0.0.1, not X.X.X.X
(X.X.X.X is host ip)

What did you expect to see?
the kube-scheduler target in prometheus is up, and no errors.
What did you see instead? Under which circumstances?
the kube-scheduler target generate a error:
Get https://X.X.X.X:10259/metrics: x509: certificate is valid for 127.0.0.1, not X.X.X.X
(X.X.X.X is host ip)
Environment
kubernetes v1.13.2

• System information:
  Linux 3.10.0-862.11.6.el7.x86_64 x86_64

• Prometheus version:
  prometheus, version 2.2.1 (branch: HEAD, revision:
  bc6058c)
  build user: root@149e5b3f0829
  build date: 20180314-14:15:45
  go version: go1.10

• Alertmanager version:

• Prometheus configuration file:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: alauda-system/kube-prometheus
    prometheus_replica: prometheus-kube-prometheus-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - kubernetes_sd_configs:
    - api_server: null
      role: endpoints
      namespaces:
        names:
        - alauda-system
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: kube-prometheus-alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http
      replacement: $1
      action: keep
rule_files:
- /etc/prometheus/rules/*.yaml
scrape_configs:
- job_name: alauda-system/kube-prometheus-exporter-kube-scheduler/0
  honor_labels: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /etc/prometheus/secrets/pki-ca/ca.crt
    cert_file: /etc/prometheus/secrets/pki-ca/scheduler.client-certificate.crt
    key_file: /etc/prometheus/secrets/pki-ca/scheduler.client-key.key
    insecure_skip_verify: false
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: exporter-kube-scheduler
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: kube-scheduler
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace

• Alertmanager configuration file:

• Logs:
  no error in prometheus log

[szediktam]

@fabxc @brancz Have a look at this issue. Thanks.

[simonpasquier]

This issue relates to the Prometheus operator rather than Proemtheus itself. I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.