New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to scrape CAdvsior in Kubernetes 1.8 #89

Closed
gaya3chandran1 opened this Issue Nov 7, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@gaya3chandran1
Copy link

gaya3chandran1 commented Nov 7, 2017

How to scrape CAdvsior in Kubernetes 1.8 . The default configuraion is unable to scarep it.

I see this in the official doc.How to modify this for this project.

  - job_name: 'kubernetes-nodes-cadvisor'
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
     - role: node
    relabel_configs:
    - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:4194/proxy/metrics/cadvisor
@chenww

This comment has been minimized.

Copy link

chenww commented Nov 9, 2017

same. Please help

@chenww

This comment has been minimized.

Copy link

chenww commented Nov 9, 2017

After I added nodes/proxy into the RBAC resource list, I could get memory and disk now, but not CPU and network. still digging

@gaya3chandran1

This comment has been minimized.

Copy link

gaya3chandran1 commented Nov 10, 2017

@chenww

This comment has been minimized.

Copy link

chenww commented Nov 10, 2017

sure thing. I got everything to work now with same job as you pasted above, but made 2 changes in different places:

  1. scrape_interval: 30s the interval has to be less than 1 minute; otherwise rate metrics will fail
  2. The ClusterRole bound to prometheus-k8s account should have nodes/proxy resource in the list if RBAC is enabled. Below is part of my configure:
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
@gaya3chandran1

This comment has been minimized.

Copy link

gaya3chandran1 commented Nov 12, 2017

Hey thanks for your suggestion added nodes/proxy in ClusterRoIe. I could get get CPU/Memory in Usage in cluster. I don't get cluster File System Usage.
I am using this dashboard https://grafana.com/dashboards/315. I get Cluster File system usage as NA.
I understand this has to be modified with server names , device=~"^/dev/[vs]da9$" but not sure how to do it in json, how to add server names there

@like-inspur

This comment has been minimized.

Copy link

like-inspur commented Nov 22, 2017

I also meeth this problem on kubernetes 1.8,with prometheus configmap add cadvisor job like this:

  • job_name: 'kubernetes-cadvisor'
    metrics_path: /metrics/cadvisor
    tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
  • role: node
    relabel_configs:
  • source_labels: [address]
    regex: '(.*):10250'
    replacement: '${1}:10255'
    target_label: address

My Environment:
kubernetes v1.8.2+coreos.0 with RBAC
prometheus v1.7.1
docker 1.13.1
Centos 7.3.1611

Problem Picture:
1
2

@maheshmadpathi

This comment has been minimized.

Copy link

maheshmadpathi commented Jul 20, 2018

I had the same issue as the kubernetes.default.svc couldn't resolve and I had to use payload IPs, plain http connection with port 10255, see below configuration for Prometheus.

- job_name: 'kubernetes-cadvisor'
   metrics_path: "/metrics/cadvisor"
   static_configs:
      - targets:
           - xxx.xx.xx.xxx:10255
           - xxx.xx.xx.xxx:10255
           - xxx.xx.xx.xxx:10255
           - xxx.xx.xx.xxx:10255



@pipo02mix

This comment has been minimized.

Copy link
Contributor

pipo02mix commented Nov 7, 2018

Done #73

@pipo02mix pipo02mix closed this Nov 7, 2018

@ghost

This comment has been minimized.

Copy link

ghost commented Nov 22, 2018

@maheshmadpathi can u share prometheus.yaml for k8s,, i am finding it very difficult to run prometheus from outside and pull k8s metrics

@pipo02mix

This comment has been minimized.

Copy link
Contributor

pipo02mix commented Nov 22, 2018

If you want to scrape the cadvisor from outside the cluster you would need to authenticate against API (certs or token). Then configure the job to scrape from the nodes proxy. Example

    - job_name: cluster-cadvisor
      scheme: https
      kubernetes_sd_configs:
      - api_server: https://<API_URL>
        role: node
        tls_config:
          ca_file: /certs/ca.pem
          cert_file: /certs/crt.pem
          key_file: /certs/key.pem
          insecure_skip_verify: false
        namespaces:
          names: []
      tls_config:
        ca_file: /certs/ca.pem
        cert_file: /certs/crt.pem
        key_file: /certs/key.pem
        insecure_skip_verify: false
      relabel_configs:
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor

You need to replace <API_URL> with the URL of your Kubernetes API and set the certs for authentication.
Feel free to make a PR with an external configuration

@ghost

This comment has been minimized.

Copy link

ghost commented Nov 22, 2018

@pipo02mix thanks, can u explain how can i get the,

tls_config:
ca_file: /certs/ca.pem
cert_file: /certs/crt.pem
key_file: /certs/key.pem

and in the api-url do i need to pass on port number,,

sorry for asking such a noob thing,, i am new to this..

@pipo02mix

This comment has been minimized.

Copy link
Contributor

pipo02mix commented Nov 22, 2018

The certs depend on where you are running k8s cluster, but in the kubeconfig you should see those or a path to them. Ideally, you should get new certs only for Prometheus and only give rights (RBAC) to scrape the nodes. Example role you can attach to those certs

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  verbs: ["get", "list", "watch"]

and in the api-url do i need to pass on port number,,

No because the default port is 443 for https

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment