Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kube-state-metrics not showing any metrics in datadog. #3275

Closed
tizzythegrey opened this issue Mar 21, 2017 · 8 comments
Closed

Kube-state-metrics not showing any metrics in datadog. #3275

tizzythegrey opened this issue Mar 21, 2017 · 8 comments

Comments

@tizzythegrey
Copy link

@tizzythegrey tizzythegrey commented Mar 21, 2017

I'm not sure what i am doing wrong here. I need to get additional metrics like pod restarts e.t.c using kube-state-metrics project but for whatever reason it is not showing in datadog metrics page. I can curl the metrics pointing to my kubernetes api-URL/metrics but the metrics are not showing in datadog. Datadog and kubernetes are integrated and i can see other default metrics but not the kube-state-metrics.

dd-agent pod, kube-state-metrics pod, kube-state-metrics service, dd-agent service and kubernetes api service are in the same cluster and can talk to each other.

Kubernetes version - 1.5.2
dd-agent version - 5.11.2

Is my version of dd-agent not compatible with kube-state-metric from https://github.com/kubernetes/kube-state-metrics? thanks for any help offered here

@hkaj
Copy link
Member

@hkaj hkaj commented Mar 22, 2017

Hi @tizzythegrey thanks for the report.

The agent has a check for kube-state-metrics. It needs to run on the same node as the kube-state-metric pod though. Otherwise you will need to specify in the kubernetes_state config file the address of the kube-state api.

Do you have an agent running on the same node? Does it have service discovery enabled (it would enable the check automatically)? If not did you configure the check explicitly (in which case you can write the config to a gist and I can have a look at it)?

@tizzythegrey
Copy link
Author

@tizzythegrey tizzythegrey commented Mar 22, 2017

Thanks @hkaj I do not have service discovery enabled so i went ahead and hard coded the address in kubernetes_state config. Yet it is not pulling the metrics into data dog.

I do not have the agents running on the same node. How do i configure the check explicitly?

@hkaj
Copy link
Member

@hkaj hkaj commented Mar 22, 2017

I think it might just be an issue with the file you modified. Could you reach out to our support team with a flare please? That will send your agent configuration (minus credentials) to us and we'll troubleshoot the issue.

@tizzythegrey
Copy link
Author

@tizzythegrey tizzythegrey commented Apr 14, 2017

This was due to the way ddagent was setup. FIxed it and we can see the metrics

@skolodyazhnyy
Copy link

@skolodyazhnyy skolodyazhnyy commented Jun 6, 2017

@tizzythegrey so, what was it? Does not work for me either :(

@tizzythegrey
Copy link
Author

@tizzythegrey tizzythegrey commented Jun 13, 2017

@skolodyazhnyy It still did not work as expected after setting it up correctly. What ended up fixing it was rolling out version 5.12.0 of dd-agent. That made all the metrics show up as expected.

@imsky
Copy link

@imsky imsky commented Aug 1, 2017

ran into the same issue today (all of a sudden losing kubernetes state metrics for most deployments). rolling out latest dd-agent did not fix it. redeployed kube-state-metrics as a DaemonSet and metrics have been restored.

cheers @tizzythegrey @hkaj @skolodyazhnyy

@kwerey
Copy link

@kwerey kwerey commented Mar 28, 2018

I'm also hitting this, having followed the instructions https://docs.datadoghq.com/integrations/kubernetes/#setup-kubernetes-state where there's no mention of running the agent on the same node as kube-state-metrics. Seems like a readme update could do some good here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants