duplicated metrics using kube-state-metrics sharding with --use-apiserver-cache

[m-messiah]

What happened:
Running kube-state-metrics in shards with enabled --use-apiserver-cache resulted in having constantly growing metrics which are not related to actual data.

What you expected to happen:
Correct sharding with reduced apiserver latency

How to reproduce it (as minimally and precisely as possible):

1. Start more than 1 shard of kube-state-metrics.
   deployment0: --shard=0 - --total-shards=2 --use-apiserver-cache
   deployment1: --shard=1 - --total-shards=2 --use-apiserver-cache
2. Scrape through servicemonitor from examples.
3. Continuously start and stop some deployments.
4. Check count by (pod) (kube_pod_info) in prometheus

Anything else we need to know?:
When I removed the flag --use-apiserver-cache the counters dropped to good values.

Probably related to #1166

Environment:

• kube-state-metrics version: 2.3.0
• Kubernetes version (use kubectl version): v1.19.13

[dgrisonnet]

This might be because the mechanism we rely on to serve cached data from the apiserver is vulnerable to stale reads: kubernetes/kubernetes#59848. I think it is not emphasized enough in the help text that this flag should be used with care because of that.

That said, do you know if this is replicable with only one instance of kube-state-metrics? Because it looks like there are way more stale data then what I would have expected.

[jpdstan]

We're seeing a similar thing here #1569 although the number of metrics is not growing unboundedly like you're seeing.

[dgrisonnet]

@jpdstan you were seeing the same problem without sharding right?

[jpdstan]

@dgrisonnet this is with sharding as well. 10 pods, approx 500k unique metrics each

[fpetkovski]

I wonder if we have a bug in https://github.com/kubernetes/kube-state-metrics/blob/main/pkg/sharding/listwatch.go.

Does the issue happen without sharding?

[JohnRusk]

@dgrisonnet

This might be because the mechanism we rely on to serve cached data from the apiserver is vulnerable to stale reads: kubernetes/kubernetes#59848.

I see that K8s issue has been recently updated to say that the problem is solved for cases where the client component (e.g. kube-state-metrics) stays running. E.g. if it stays running but fails over to a different API server instance then the problem won't happen (as long as it uses the Informer class).

But if the component itself restarts, the problem may still happen.

Does that recent update change the seriousness of the stale read problem, for kube-state-metrics?

[e-ngo]

@fpetkovski
Does anyone have the context as to why we set RV = 0 in the Watch method despite the Reflector setting this RV For us after the initial List?
Code for reflector: https://github.com/kubernetes/client-go/blob/v0.22.0/tools/cache/reflector.go#L401-L415
Offending lines:

kube-state-metrics/pkg/watch/watch.go

Lines 97 to 99 in 008bdb1

	if i.useAPIServerCache {
	options.ResourceVersion = "0"
	}

I feel like this is a bug.

[dgrisonnet]

@JohnRusk I am not sure I understand your point. If --use-apiserver-cache is enabled, kube-state-metrics will use the client in a way where it will automatically fetch data from the apiserver cache instead of requiring a quorum read in etcd. Whether kube-state-metrics restarts or not, nothing will change since the cache is on the kube-apiserver side.

[dgrisonnet]

@e-ngo this indeed seems like a bug, good catch

[dgrisonnet]

Could any of you perhaps try to see if the bug still exists after the fix made by @e-ngo? This image contains it: https://console.cloud.google.com/gcr/images/k8s-staging-kube-state-metrics/global/kube-state-metrics@sha256:126f7ef47ac7723b19cc9bc6a3d63c71bcd87888cd4c12e0101684a2eb7ca804/details

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[m-messiah]

/remove-lifecycle stale

[jpdstan]

i've tried the patch from @e-ngo , and using the query count by (cluster_name, namespace, pod) (kube_pod_info{pod_ip=~".+"}) > 1, it seems like the stale metrics are not happening anymore! i think we can close out this issue.

[mrueg]

Thanks for the feedback!