fix: fix the issue that duplicated CR metrics are generated when CRD is deleted/applied multiple times

[CatherineF-dev]

What this PR does / why we need it: #2223

# It returns same metrics data many times when CRD is deleted/applied multiple times
 curl localhost:8089/metrics
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Mitigation for #2223

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: CatherineF-dev
Once this PR has been reviewed and has the lgtm label, please assign mrueg for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mrueg]

Does it make sense to have a map[string] bool here?

[CatherineF-dev]

I think using unique is fine as a mitigation.

It didn't remove deleted CR https://github.com/rexagod/kube-state-metrics/blob/25a1d8da057cf761d614c59a52785335d34082d1/internal/discovery/discovery.go#L232. After this is fixed, we can remove unique.

[dgrisonnet]

I don't think this will fix the inherent issue. It seems that the mechanism detecting the deletion of CRD matched by the ksm config is not working properly. Normally when that happens, the internal store should be deleted alongside its informer because we don't want to expose metrics about this resource anymore.

[dgrisonnet]

To be clearer, the scenario where a specific resource appears twice in ksm code should never happen, so we should understand why it happened and fix it instead of enforcing uniqueness.

[CatherineF-dev]

got it, agree. Will debug again to find root cause.

[CatherineF-dev]

It should be related to #1851 where adding PollForCacheUpdates function.

[CatherineF-dev]

It deletes removed CR here https://github.com/rexagod/kube-state-metrics/blob/main/internal/discovery/discovery.go#L42C24-L116, while didn't delete removed CR here https://github.com/rexagod/kube-state-metrics/blob/25a1d8da057cf761d614c59a52785335d34082d1/internal/discovery/discovery.go#L232.

How about creating a new issue and merging this PR as a mitigation?
I think it needs some changes around #1851.

Or we only want the fix PR instead of mitigation.

cc @dgrisonnet

[logicalhan]

/triage accepted
/assign @dgrisonnet

[rexagod]

I second #2257 (comment), and as for #2257 (comment), I'd prefer the changes in this PR (instead of mitigating the issue, we want to identify the root cause and fix that since duplicacy shouldn't ideally be exhibited at all).

[korjek]

any updates here?
It leads to losing the metrics, which is not good.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale