How to specify the host of k8s apiserver

[Kevin-P-Huang]

What did you expect to see?

Specify the host of k8s apiserver

What did you see instead? Under which circumstances?

Environment

• System information:

$ uname -srm
Linux 3.10.0-862.3.2.el7.x86_64 x86_64

• Prometheus version:

quay.io/prometheus/prometheus:v2.0.0

• Alertmanager version:

  insert output of alertmanager --version here (if relevant to the issue)

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

• Prometheus statefulset deployment file:

$  cat prometheus-k8s-statefulset.yaml
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  namespace: monitoring
  labels:
    prometheus: k8s
spec:
  replicas: 2
  version: v2.0.0
#  serviceAccountName: prometheus-k8s
  serviceMonitorSelector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
  ruleSelector:
    matchLabels:
      role: prometheus-rulefiles
      prometheus: k8s
  resources:
    requests:
      memory: 4G
  storage:
    volumeClaimTemplate:
      metadata:
        name: prometheus-data
      spec:
        selector:
          matchLabels:
            app: my-example-prometheus
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 40Gi
  alerting:
    alertmanagers:
    - namespace: monitoring
      name: alertmanager-main
      port: web

To simplify installation, TLS Certificates is not used in our k8s environment( 3 master and 5 node in cluster), and URL of apiserver in our k8s environment is "http://192.168.1.4:8080" and is not “https://10.10.10.1:443”.

It will sometimes bring us trouble when deploy.Because some containers will automatically connect apiserver by https' URL when be deployed, and error which info like open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory occur.
For example, when we deploy prometheus in our k8s environment, error info will be printed in pod's log.

• Logs:

[root@master01 prometheus]# kubectl -n monitoring logs -f pod/prometheus-k8s-0 prometheus
level=info ts=2018-08-24T14:45:13.61526488Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2018-08-24T14:45:13.615332616Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2018-08-24T14:45:13.615364416Z caller=main.go:217 host_details="(Linux 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 prometheus-k8s-0 (none))"
level=info ts=2018-08-24T14:45:13.617453871Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-08-24T14:45:13.61754072Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2018-08-24T14:45:13.617522749Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2018-08-24T14:45:13.638411777Z caller=main.go:326 msg="TSDB started"
level=info ts=2018-08-24T14:45:13.638460763Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/config/prometheus.yaml
E0824 14:45:13.639229       1 config.go:290] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
level=info ts=2018-08-24T14:45:13.639780326Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
E0824 14:45:13.640087       1 config.go:290] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
level=info ts=2018-08-24T14:45:13.640103585Z caller=kubernetes.go:100 component=notifier discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-08-24T14:45:13.640363028Z caller=main.go:371 msg="Server is ready to receive requests."
level=error ts=2018-08-24T14:45:13.676899562Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: Get https://10.10.10.1:443/api/v1/namespaces/monitoring/services?resourceVersion=0: x509: certificate signed by unknown authority"
level=error ts=2018-08-24T14:45:13.676928324Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Pod: Get https://10.10.10.1:443/api/v1/namespaces/monitoring/pods?resourceVersion=0: x509: certificate signed by unknown authority"
level=error ts=2018-08-24T14:45:13.677110305Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: Get https://10.10.10.1:443/api/v1/namespaces/monitoring/endpoints?resourceVersion=0: x509: certificate signed by unknown authority"

To avoid connecting to the https url of apiserver, we will find the arg or env of container to specify the host of apiserver. For example, we add an env of container which name "APISERVER_HOST" to specify the host of apiserver when deploy elasticsearch of k8s addon fluentd-elasticsearch

  spec:
      containers:
      - image: k8s.gcr.io/elasticsearch:v6.2.4
        name: elasticsearch-logging
        resources:
          limits:
            cpu: 1000m
        [...]
        env:
        - name: APISERVER_HOST
          value: "http://192.168.1.4:8080"

[simonpasquier]

You can use the api_server key in the Kubernetes SD configuration to pass the URL of the API server. From the docs:

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.