[prometheus-kube-stack] labels exported_namespace and exported_pod instead of namespace, pod in prometheus

[broken-ufa]

Describe the bug a clear and concise description of what the bug is.

After upgrade from chart 16.9.1 to 18.0.12 grafana workload dashboards were broken

Problem in writing kube-state-metrics metrics to prometheus:
e.g.

broken@kube-master:~$ curl http://10.244.6.78:8080/metrics 2>/dev/null | grep kube_pod_container_info | grep apiserver

kube_pod_container_info{namespace="kube-system",pod="kube-apiserver-kube-master",uid="1a28c881-28ca-44d4-bb52-92b945886a83",container="kube-apiserver",image="k8s.gcr.io/kube-apiserver:v1.21.2",image_id="106ff58d4308243e0042862435f5a0b14dd332d8151f17a739046c7df33c7ae6",container_id="cri-o://5486cff7360c5a8f6fb8dc5efdafced6a9a90d91595972c34e1dec32ea74f7ef"} 1

but in prometheus this metrics were writed like:

kube_pod_container_info{container="kube-state-metrics", container_id="cri-o://5486cff7360c5a8f6fb8dc5efdafced6a9a90d91595972c34e1dec32ea74f7ef", endpoint="http", exported_container="kube-apiserver", exported_namespace="kube-system", exported_pod="kube-apiserver-kube-master", image="k8s.gcr.io/kube-apiserver:v1.21.2", image_id="106ff58d4308243e0042862435f5a0b14dd332d8151f17a739046c7df33c7ae6", instance="10.244.6.78:8080", job="kube-state-metrics", namespace="monitoring", pod="prometheus-stack-kube-state-metrics-668cd64d6c-m4vlh", service="prometheus-stack-kube-state-metrics", uid="1a28c881-28ca-44d4-bb52-92b945886a83"}

labels namespace, pod, container rewrited to exported* metrics

What's your helm version?

v3.6

What's your kubectl version?

v1.21.2

Which chart?

kube-prometheus-stack

What's the chart version?

18.0.12

What happened?

All of metric from kube-state-metrics have 'namespace=monitoring' label instead of real namespace label from metrics

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

none

Anything else we need to know?

No response

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[alexku7]

faced the same issue .

Check if you have honor_labels: true
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L1326

it solved the issue for me

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.