AWS EKS: cluster doesn't provide client-ca-file

[toriacht]

Hi,

I am trying to deploy on AWS EKS which now supports Kubernetes HPA AWS Blog. I successfulyy deployed Prometheus via helm but now when i try to deploy the adapter i get the following error

`kubectl logs promadapter-prometheus-adapter-6d68c79bbc-pjgvw
I1018 11:41:13.278829 1 serving.go:273] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
Error: cluster doesn't provide client-ca-file
...
...
panic: cluster doesn't provide client-ca-file

goroutine 1 [running]:
main.main()
/go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x13b
`

How can i point it towards this ca file?
I tried pointing to .kube config but this did nothing

helm install --name custommetrics stable/prometheus-adapter --set --lister-kubeconfig=/root/.kube/config

Any tips much appreciated...

[bhavin192]

Duplicate of #108

[bhavin192]

Hey, this was fixed in #110, can you please try image with latest tag?
Or you can also try image mentioned in #108 (comment)

[twittyc]

So I tried with the latest built image, but I'm running into issues authenticating against the apiserver. Do I need to be providing the --requestheader-client-ca-file flag? I've tried with and without it with no luck. @bhavin192 Do you have an example of the config you used with your deployment? Thanks!

[bwang221]

I ran into this today. any update?

[bhavin192]

@twittyc I'm using the helm chart to install Prometheus adapter, and I'm using image with tag pr110 as mentioned here #108 (comment)

custom-values.yaml

image:
  repository: docker.io/bhavin192/k8s-prometheus-adapter-amd64
  tag: pr110

@bwang221

I ran into this today. any update?

Have you tried this #119 (comment) ?

[VinayVanama]

Hey, this was fixed in #110, can you please try image with latest tag?
Or you can also try image mentioned in #108 (comment)

Hi ! Even I'm facing with the same issue, where will be my ca.crt file located ?

[bhavin192]

@VinayVanama which version of Prometheus adapter are you using?

[Vlaaaaaaad]

Hi,

Running on the latest EKS( platform version eks.1 with k8s version v1.11.5) this still fails, even when using the latest image. I am also using SSL for Prometheus so that might also affect this.

I have an app pushing a metric to Prometheus Gateway which does send the metric forward to Prometheus( I've checked).

I am deploying using the helm chart with the following values added to the default:

prometheus:
  url: https://prometheus.example.com
  port: 443

image:
  tag: latest

Relevant logs:

I0111 13:54:22.060136       1 adapter.go:91] successfully using in-cluster auth
E0111 13:54:22.128189       1 provider.go:206] unable to update list of all metrics: unable to fetch metrics for query "{__name__=~\"^container_.*\",container_name!=\"POD\",namespace!=\"\",pod_name!=\"\"}": Get https://prometheus.example.com:443/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer_name%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod_name%21%3D%22%22%7D&start=1547213662.12: x509: failed to load system roots and no roots provided
I0111 13:54:23.284843       1 serving.go:273] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
W0111 13:54:23.729479       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0111 13:54:23.737793       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
I0111 13:54:23.742826       1 serve.go:96] Serving securely on [::]:6443
I0111 13:54:57.866291       1 wrap.go:42] GET /healthz: (9.839133ms) 200 [[kube-probe/1.11] 192.168.163.138:33770]
I0111 13:54:57.897339       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:54:57.897403       1 wrap.go:42] GET /: (1.745015ms) 403 [[Go-http-client/2.0] 192.168.63.162:52122]
I0111 13:54:57.900228       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:54:57.900282       1 wrap.go:42] GET /: (110.948µs) 403 [[Go-http-client/2.0] 192.168.98.128:45844]
I0111 13:54:57.902588       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:54:57.902637       1 wrap.go:42] GET /: (98.962µs) 403 [[Go-http-client/2.0] 192.168.63.162:52122]
I0111 13:54:57.907968       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:54:57.908017       1 wrap.go:42] GET /: (94.553µs) 403 [[Go-http-client/2.0] 192.168.98.128:45844]
I0111 13:54:57.913624       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:54:57.913670       1 wrap.go:42] GET /: (90.328µs) 403 [[Go-http-client/2.0] 192.168.98.128:45844]
I0111 13:54:58.678638       1 wrap.go:42] GET /healthz: (70.181µs) 200 [[kube-probe/1.11] 192.168.163.138:33772]
I0111 13:55:00.726537       1 wrap.go:42] GET /openapi/v2: (2.075104ms) 404 [[] 192.168.98.128:45880]
I0111 13:55:00.729392       1 wrap.go:42] GET /swagger.json: (1.687249ms) 404 [[] 192.168.98.128:45880]
I0111 13:55:00.735662       1 wrap.go:42] GET /openapi/v2: (158.459µs) 404 [[] 192.168.63.162:52168]
I0111 13:55:00.736832       1 wrap.go:42] GET /swagger.json: (182.868µs) 404 [[] 192.168.63.162:52168]
I0111 13:55:01.825191       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (269.389µs) 200 [[kube-apiserver/v1.11.5 (linux/amd64) kubernetes/6bad6d9] 192.168.63.162:52168]
I0111 13:55:07.856627       1 wrap.go:42] GET /healthz: (86.212µs) 200 [[kube-probe/1.11] 192.168.163.138:33796]
I0111 13:55:08.690186       1 wrap.go:42] GET /healthz: (10.840672ms) 200 [[kube-probe/1.11] 192.168.163.138:33798]
I0111 13:55:10.035229       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (277.628µs) 200 [[kube-controller-manager/v1.11.5 (linux/amd64) kubernetes/6bad6d9/generic-garbage-collector] 192.168.63.162:52168]
I0111 13:55:12.360628       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/myService/myTaskCount: (416.913µs) 404 [[kube-controller-manager/v1.11.5 (linux/amd64) kubernetes/6bad6d9/horizontal-pod-autoscaler] 192.168.63.162:52168]
I0111 13:55:12.962239       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:55:12.962313       1 wrap.go:42] GET /: (1.906976ms) 403 [[Go-http-client/2.0] 192.168.98.128:45844]
I0111 13:55:17.856421       1 wrap.go:42] GET /healthz: (93.821µs) 200 [[kube-probe/1.11] 192.168.163.138:33840]
I0111 13:55:18.678648       1 wrap.go:42] GET /healthz: (84.844µs) 200 [[kube-probe/1.11] 192.168.163.138:33842]
I0111 13:55:20.581032       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.113µs) 200 [[kube-apiserver/v1.11.5 (linux/amd64) kubernetes/6bad6d9] 192.168.98.128:45880]
E0111 13:55:22.142904       1 provider.go:206] unable to update list of all metrics: unable to fetch metrics for query "{__name__=~\"^container_.*\",container_name!=\"POD\",namespace!=\"\",pod_name!=\"\"}": Get https://prometheus.example.com:443/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer_name%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod_name%21%3D%22%22%7D&start=1547213722.128: x509: failed to load system roots and no roots provided
I0111 13:55:23.561683       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (340.508µs) 200 [[kube-controller-manager/v1.11.5 (linux/amd64) kubernetes/6bad6d9/controller-discovery] 192.168.63.162:52168]
I0111 13:55:24.471063       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (260.147µs) 200 [[kube-controller-manager/v1.11.5 (linux/amd64) kubernetes/6bad6d9/resourcequota-controller] 192.168.63.162:52168]
I0111 13:55:24.574581       1 authorization.go:73] Forbidden: "/", Reason: ""
I0111 13:55:24.574645       1 wrap.go:42] GET /: (9.980898ms) 403 [[Go-http-client/2.0] 192.168.63.162:52122]
I0111 13:55:27.858330       1 wrap.go:42] GET /healthz: (1.93816ms) 200 [[kube-probe/1.11] 192.168.163.138:33870]
I0111 13:55:28.678671       1 wrap.go:42] GET /healthz: (91.654µs) 200 [[kube-probe/1.11] 192.168.163.138:33872]

Let me know if I am doing anything wrong or you'd like me to investigate some more.

[Vlaaaaaaad]

Welp, in a surprising twist, switching from HTTPS Ingress to using http://prometheus-pushgateway.monitoring.svc.cluster.local:9091makes it all work.

Relevant logs:

I0116 15:01:39.390164       1 adapter.go:91] successfully using in-cluster auth
I0116 15:01:39.425981       1 request.go:1099] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0116 15:01:40.057777       1 serving.go:273] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
W0116 15:01:40.720654       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0116 15:01:40.727922       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
I0116 15:01:40.732466       1 healthz.go:83] Installing healthz checkers:"ping"
I0116 15:01:40.732592       1 serve.go:96] Serving securely on [::]:6443

No idea why it would fail with Ingress but work with cluster.local services. I imagine I was somwhoe configuring the Ingress badly? Everything else seemed to work with the Ingress tho. I am confused.

Either way, my apologies.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.