After upgrade to kubernetes 1.11 : Unable to detect the kubelet URL automatically.

[ohardy]

[ AGENT ] 2018-06-29 13:34:00 UTC | ERROR | (runner.go:277 in work) | Error running check kubelet: [{"message": "Unable to detect the kubelet URL automatically.", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/base.py\", line 294, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubelet/kubelet.py\", line 86, in check\n    raise CheckException(\"Unable to detect the kubelet URL automatically.\")\nCheckException: Unable to detect the kubelet URL automatically.\n"}]
[ AGENT ] 2018-06-29 13:34:08 UTC | ERROR | (autoconfig.go:514 in collect) | Unable to collect configurations from provider Kubernetes pod annotation: permanent failure in kubeutil: retry number exceeded
[ AGENT ] 2018-06-29 13:34:15 UTC | ERROR | (kubeutil.go:50 in GetKubeletConnectionInfo) | connection to kubelet failed: permanent failure in kubeutil: retry number exceeded

Any ideas ?

[JulienBalestra]

@ohardy thank you for submitting this issue.

Can you provide us more details about your setup?
Is it possible to give us a flare through a support ticket?

[sciyoshi]

I had this issue as well after building a cluster on k8s 1.11 with kubeadm, for me it seemed like kubelet no longer listens on port 10255 (the ReadOnlyPort) by default. I fixed this by adding adding KUBELET_EXTRA_ARGS=--read-only-port=10255 in /etc/default/kubelet on the node host and restarting the kubelet service.

[sciyoshi]

FWIW - the full error logs that I was seeing:

[ AGENT ] 2018-07-09 22:22:53 UTC | ERROR | (autoconfig.go:530 in collect) | Unable to collect configurations from provider Kubernetes pod annotation: temporary failure in kubeutil, will retry later: cannot connect: https: "Get https://172.20.61.34:10250/pods: x509: cannot validate certificate for 172.20.61.34 because it doesn't contain any IP SANs", http: "Get http://172.20.61.34:10255/pods: dial tcp 172.20.61.34:10255: connect: connection refused"
[ AGENT ] 2018-07-09 22:22:56 UTC | ERROR | (kubeutil.go:50 in GetKubeletConnectionInfo) | connection to kubelet failed: temporary failure in kubeutil, will retry later: try delay not elapsed yet

[JulienBalestra]

@sciyoshi it seems the given Kubelet host to the datadog agent doesn't match your kubelet SAN.
The value of the kubernetes_kubelet_host is 172.20.61.34 but your cert is issued with something else.

Can you provide more details about your setup?
Like:

• openssl x509 -in /path/to/kubelet/server/cert -text (default in /var/lib/kubelet/pki)

[eduardohl]

Having the same problems here, it's a Rancher Kubernetes installation...

[sukrit007]

Just ran into this issue with Rancher Kubernetes. Wondering if kubelet_tls_verify can fix it and how can we set this parameter in docker agent.

[Octopixell]

Ran into the same issue and @sciyoshi solution seems to have fixed the issue for me as well. Thanks!

[mfpierre]

Closing the issue as on Rancher setups you have to enable the read only port as documented in https://rancher.com/blog/2018/2018-07-16-how-to-deploy-datadog/

[vmendi]

The read only port seems to be deprecated by the Kubernetes project:

kubernetes-sigs/kubespray#2159
kubernetes/kubeadm#732

Is there a solution that doesn't involve opening this port?

[Aiqbal1234]

@vmendi Any follow up on this?

[shcallaway]

For anyone using kubeadm, this is from the Kubernetes 1.11 release notes:

kubelets in kubeadm clusters now disable the readonly port (10255). If you're relying on unauthenticated access to the readonly port, please switch to using the secure port (10250).

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#before-upgrading

[shcallaway]

Confusingly, even though the read only port has been supposedly disabled, the Kubernetes docs say that it defaults to 10255:

The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255)

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/