Kube-state-metrics not showing any metrics in datadog.

[tizzythegrey]

I'm not sure what i am doing wrong here. I need to get additional metrics like pod restarts e.t.c using kube-state-metrics project but for whatever reason it is not showing in datadog metrics page. I can curl the metrics pointing to my kubernetes api-URL/metrics but the metrics are not showing in datadog. Datadog and kubernetes are integrated and i can see other default metrics but not the kube-state-metrics.

dd-agent pod, kube-state-metrics pod, kube-state-metrics service, dd-agent service and kubernetes api service are in the same cluster and can talk to each other.

Kubernetes version - 1.5.2
dd-agent version - 5.11.2

Is my version of dd-agent not compatible with kube-state-metric from https://github.com/kubernetes/kube-state-metrics? thanks for any help offered here

[hkaj]

Hi @tizzythegrey thanks for the report.

The agent has a check for kube-state-metrics. It needs to run on the same node as the kube-state-metric pod though. Otherwise you will need to specify in the kubernetes_state config file the address of the kube-state api.

Do you have an agent running on the same node? Does it have service discovery enabled (it would enable the check automatically)? If not did you configure the check explicitly (in which case you can write the config to a gist and I can have a look at it)?

[tizzythegrey]

Thanks @hkaj I do not have service discovery enabled so i went ahead and hard coded the address in kubernetes_state config. Yet it is not pulling the metrics into data dog.

I do not have the agents running on the same node. How do i configure the check explicitly?

[hkaj]

I think it might just be an issue with the file you modified. Could you reach out to our support team with a flare please? That will send your agent configuration (minus credentials) to us and we'll troubleshoot the issue.

[tizzythegrey]

This was due to the way ddagent was setup. FIxed it and we can see the metrics

[skolodyazhnyy]

@tizzythegrey so, what was it? Does not work for me either :(

[tizzythegrey]

@skolodyazhnyy It still did not work as expected after setting it up correctly. What ended up fixing it was rolling out version 5.12.0 of dd-agent. That made all the metrics show up as expected.

[imsky]

ran into the same issue today (all of a sudden losing kubernetes state metrics for most deployments). rolling out latest dd-agent did not fix it. redeployed kube-state-metrics as a DaemonSet and metrics have been restored.

cheers @tizzythegrey @hkaj @skolodyazhnyy

[kwerey]

I'm also hitting this, having followed the instructions https://docs.datadoghq.com/integrations/kubernetes/#setup-kubernetes-state where there's no mention of running the agent on the same node as kube-state-metrics. Seems like a readme update could do some good here.