[stable/prometheus] How to monitor etcd with ssl?

[minhdanh]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Version of Helm and Kubernetes:

helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart: stable/prometheus

What happened:
I need to update prometheus to scrape the etcd servers of the same cluster that Prometheus is running on. Here's my job config:

      - job_name: 'kubernetes-etcd'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
          - targets: ['10.6.16.99:2379','10.6.2.41:2379','10.6.18.105:2379']

This doens't work as the cert file (ca.crt) is for Kubernetes cluster, not for etcd cluster.
Prometheus complains of bad certificate when scraping these endpoints:

Get https://10.6.16.99:2379/metrics: remote error: tls: bad certificate

What you expected to happen:
Is there a way to add these cert files that is needed for Prometheus to scrape etcd?

I imaging something like some variables with contents to mount to prometheus. Such as:

additionalFiles:
   etcd_ca_file: |
     [content of ca.crt]
   etcd_cert_file: |
     [...]
   ...

Then those file will be mounted to Prometheus in /prometheus.

Then the job configs for etcd can refer to these files.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

[Nodraak]

I ended up manually loading etcd's certificates into a Kubernetes Secret and adding this to prometheus chart's config :

  extraSecretMounts:
    - name: prometheus-etcd-certificates
      mountPath: /etcd-ssl
      secretName: prometheus-etcd-certificates
      readOnly: true

[...]
      - job_name: 'etcd'
        [...]
        tls_config:
          insecure_skip_verify: true
          cert_file: /etcd-ssl/tls.crt
          key_file: /etcd-ssl/tls.key

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[aleks-mariusz]

in case this helps anyone, i built an on-prem v1.13.5 kubernetes cluster using kubeadm..

i also deploy my prometheus-operator into its own "metrics" namespace.

so during cluster initialization, while bootstrapping the cluster, i also capture the generated etcd certs on the first master node using:

D="$(mktemp -d)"
cp /etc/kubernetes/pki/etcd/{ca.crt,healthcheck-client.{crt,key}} $D
kubectl create ns metrics
kubectl -n metrics create secret generic etcd-client --from-file="$D"
rm -fr "$D"

and then adding this to my values.yaml:

kubeEtcd:
  serviceMonitor:
    scheme: https
    insecureSkipVerify: true
    caFile: /etc/prometheus/secrets/etcd-client/ca.crt
    certFile: /etc/prometheus/secrets/etcd-client/healthcheck-client.crt
    keyFile: /etc/prometheus/secrets/etcd-client/healthcheck-client.key

prometheus:
  prometheusSpec:
    secrets:
      - etcd-client

then the target gets scraped properly

[caihy05]

挂载证书解决问题

[caihy05]

第一步：
kubectl create secret generic prome-etcd-cert --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem -n monitoring
第二步：
volumeMounts:
- name: ssl
mountPath: /ssl
volumes:
- name: ssl
secret:
secretName: prome-etcd-cert

第三步：
- job_name: 'etcd'
scheme: https
tls_config:
insecure_skip_verify: true
cert_file: /ssl/etcd.pem
key_file: /ssl/etcd-key.pem
static_configs:
- targets: ['172.16.66.89:2379','172.16.66.90:2379','172.16.66.91:2379']