Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High cardinality metrics coming from reflectors #52121

Closed
smarterclayton opened this issue Sep 7, 2017 · 4 comments · Fixed by #54921
Closed

High cardinality metrics coming from reflectors #52121

smarterclayton opened this issue Sep 7, 2017 · 4 comments · Fixed by #54921
Labels
sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation.

Comments

@smarterclayton
Copy link
Contributor

smarterclayton commented Sep 7, 2017 •
edited

I'm seeing a large number of unique prometheus series showing up (due to the logging we added for anonymous reflectors).


reflector_storage_cacher_go:_registry_sdnnetnamespaces_2264_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_registry_sdnnetworks_2265_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_registry_sdnsubnets_2263_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_replicasets_2299_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1130133
reflector_storage_cacher_go:_resourcequotas_2278_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_rolebindingrestrictions_2275_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_rolebindings_2304_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1250058
reflector_storage_cacher_go:_roles_2303_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_routes_2269_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1250034
reflector_storage_cacher_go:_secrets_2279_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1250056
reflector_storage_cacher_go:_securitycontextconstraints_2253_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_storage_cacher_go:_serviceaccounts_2280_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"}


reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_11609_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-8",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-n-kkhm",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-n-kkhm",role="app"} | 1250064
reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_6999_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-2",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-m-11v4",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-m-11v4",role="infra",subrole="master"} | 1250064
reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_9871_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-8",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-n-kwtq",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-n-kwtq",role="app"} | 1250064
reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:404_11610_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-8",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-n-kkhm",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-n-kkhm",role="app"} | 1313098
reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:404_7000_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-2",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-m-11v4",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-m-11v4",role="infra",subrole="master"}




reflector_github_com_openshift_origin_pkg_build_generated_informers_internalversion_factory_go:45_664_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1131114
reflector_github_com_openshift_origin_pkg_build_generated_informers_internalversion_factory_go:45_671_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1131132
reflector_github_com_openshift_origin_pkg_deploy_generated_informers_internalversion_factory_go:45_662_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1250168
reflector_github_com_openshift_origin_pkg_image_generated_informers_internalversion_factory_go:45_2331_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1131058
reflector_github_com_openshift_origin_pkg_image_generated_informers_internalversion_factory_go:45_672_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1131058
reflector_github_com_openshift_origin_pkg_quota_generated_informers_internalversion_factory_go:45_2332_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_github_com_openshift_origin_pkg_quota_generated_informers_internalversion_factory_go:45_673_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_11611_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-8",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-n-kkhm",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-n-kkhm",role="app"} | 1124352
reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_634_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_7001_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-2",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-m-11v4",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-m-11v4",role="infra",subrole="master"} | 1124352
reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_9873_last_resource_version{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="n1-standard-8",beta_kubernetes_io_os="linux",failure_domain_beta_kubernetes_io_region="us-central1",failure_domain_beta_kubernetes_io_zone="us-central1-a",instance="origin-ci-ig-n-kwtq",job="kubernetes-nodes",kubernetes_io_hostname="origin-ci-ig-n-kwtq",role="app"} | 1124352
reflector_github_com_openshift_origin_pkg_security_generated_informers_internalversion_factory_go:45_2333_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_github_com_openshift_origin_pkg_security_generated_informers_internalversion_factory_go:45_674_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_serviceaccounts_controllers_docker_registry_service_go:123_633_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1250064
reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_635_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_675_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_676_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1124352
reflector_github_com_openshift_origin_pkg_unidling_controller_unidling_controller_go:195_677_last_resource_version{instance="10.128.0.2:8444",job="kubernetes-controllers"} | 1311851
reflector_github_com_openshift_origin_pkg_user_generated_informers_internalversion_factory_go:45_2334_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_github_com_openshift_origin_vendor_k8s_io_apiextensions_apiserver_pkg_client_informers_internalversion_factory_go:61_2314_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1124352
reflector_github_com_openshift_origin_vendor_k8s_io_client_go_informers_factory_go:72_2313_last_resource_version{instance="10.128.0.2:8443",job="kubernetes-apiservers"} | 1313170

This is a small cluster but I believe this may have unbounded cardinality.
@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 7, 2017
@smarterclayton smarterclayton added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Sep 7, 2017
@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 7, 2017
@lavalamp
Copy link
Member

...are we storing the resource version in the metric name?

@lavalamp
Copy link
Member

cc @mml

@smarterclayton smarterclayton mentioned this issue Oct 6, 2017
Closed
@sttts sttts mentioned this issue Oct 10, 2017
Merged
@weiwei04
Copy link
Contributor

weiwei04 commented Nov 1, 2017 •
edited

Came into the same problem, checked the code, seems we use reflector + reflector_name + auto incrementing as metric names, for most reflector, they use a default name provided by getDefaultReflectorName which is go filename.

I think the proper way is use reflector_last_resource_version as the metric name, and add a reflector_name label, like

reflector_last_resource_version{instance="104.154.20.21:443",job="kubernetes-apiservers",name="storage_cacher_go_registry_sdnnetnamespaces_2264"}

@smarterclayton @lavalamp

@weiwei04 weiwei04 mentioned this issue Nov 1, 2017
Merged
@weiwei04
Copy link
Contributor

weiwei04 commented Nov 1, 2017

opened a pr to fix this, ptal @smarterclayton @lavalamp thanks :)

k8s-github-robot pushed a commit that referenced this issue Nov 2, 2017
Merge pull request #54921 from weiwei04/fix_reflector_last_resource_v…
b7968e0 
…ersion

Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

rename metric reflector_xx_last_resource_version

**What this PR does / why we need it**:

mv reflector name from metric name to metric label

before:

```
reflector_k8s_io_kubernetes_pkg_client_informers_informers_generated_internalversion_factory_go:73_8664_last_resource_version{instance="104.154.20.21:443",job="kubernetes-apiservers"}
```

after

```
reflector_last_resource_version{instance="10.0.2.15:6443",job="kubernetes-apiservers",name="k8s_io_kubernetes_pkg_client_informers_informers_generated_internalversion_factory_go_73_8664"}
```

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #52121 

**Special notes for your reviewer**:

None

**Release note**:

```release-note
NONE
```
@discordianfish discordianfish mentioned this issue Nov 15, 2017
Closed
@szuecs szuecs mentioned this issue Feb 1, 2019
Closed
@logicalhan logicalhan mentioned this issue Feb 2, 2023
Merged
@logicalhan logicalhan mentioned this issue Feb 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rename metric reflector_xx_last_resource_version weiwei04/kubernetes
4 participants
@lavalamp @smarterclayton @weiwei04 @k8s-github-robot