kubernetes-cadvisor giving no token found

[burizz]

Hello,

I have a Kubernetes Cluster in Google Cloud trying install Prometheus on it. But I hit a strange issue. The kubernetes-cadvisor component is showing as down for all nodes. I am not sure what I am missing here or if it actually is a bug. Any suggestion would be great. You can find my yaml configs below :

Kubernetes Cluster in Google CLoud v1.9.3-gke.0

Here is my configmap configuration for it :

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}:4194/proxy/metrics/cadvisor

I did a lot of searching and tried many variations (with and without port on the replacement: field, with and without TLS, etc) but it seems like a bug unless I am missing something.

When I remove the port from - replacement: /api/v1/nodes/${1}:4194/proxy/metrics/cadvisor I get 404 not found. With the port I get token not found.

I've also used the follow RBAC configuration :

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: prometheus
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  labels:
    app: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

And my deployment yaml :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v2.1.0
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          emptyDir: {}

Service :

---

apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
name: prometheus-service
kubernetes.io/name: "Prometheus"
name: prometheus-service
spec:
selector:
app: prometheus
type: LoadBalancer
ports:
- port: 8080
targetPort: 9090
protocol: TCP
loadBalancerSourceRanges:
- 0.0.0.0/32 (anonymized)
- 0.0.0.0/23 (anonymized)

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.