Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lens Metrics get wrong values of CPU, RAM #7299

Closed
karkill opened this issue Mar 7, 2023 · 24 comments
Closed

Lens Metrics get wrong values of CPU, RAM #7299

karkill opened this issue Mar 7, 2023 · 24 comments
Labels
area/metrics All the things related to metrics bug Something isn't working p1

Comments

@karkill
Copy link

karkill commented Mar 7, 2023

Describe the bug
I use the m6i.large instance (AWS), it is in the general purpose family with 2 vCPUs, 8.0 GiB RAM but in Lens I see it double (4 vCPUs, 15.265 GiB RAM)
Screenshots
image
image

Environment (please complete the following information):

  • Lens Version: 2023.3.21022-latest
  • OS: MACOS
@karkill karkill added the bug Something isn't working label Mar 7, 2023
@Nokel81
Copy link
Collaborator

Nokel81 commented Mar 7, 2023

Do you have multiple prometheus installations installed on this cluster?

@Nokel81 Nokel81 added the area/metrics All the things related to metrics label Mar 7, 2023
@karkill
Copy link
Author

karkill commented Mar 8, 2023

I have only one prometheus installed by kube-prometheus-stack

@massimeddu-sj
Copy link

massimeddu-sj commented Mar 9, 2023

Hi have a similar problem with POD memory metric.

At pod level I see the the double of the real value, while at container level I see the right value. (The pod has just one container). See attached screenshot. I also have a Prometheus stack installed with kube-prometheus-stack.

image (12)
image (13)

Lens version: 2023.3.71735-latest
OS: Linux

It looks a regression with the latest version of Lens, because with older version of Lens I see the correct value.

@kyleli666
Copy link

kyleli666 commented Mar 10, 2023

@jweak my Prometheus is bitnami/kube-prometheus chartVesion8.3.12 appVersion0.63.0
I have POD memory metric doubled, while container memory is correct. Node metrics are also correct.
My cluster is EKS v1.24
this bug happen with all deployments
the metrics were correct in Lens Extension API v6.2.5

@andrezaycev
Copy link

andrezaycev commented Mar 12, 2023

+1

image

@saad-littera
Copy link

+1

@centromere
Copy link

These are the values being summed together:

Screenshot 2023-03-22 at 12 21 47

Related code:

case "memoryUsage":
return `sum(container_memory_working_set_bytes{pod=~"${opts.pods}", namespace="${opts.namespace}"}) by (${opts.selector})`;

The problem may have been introduced in this commit.

@Nokel81
Copy link
Collaborator

Nokel81 commented Mar 22, 2023

@centromere Thanks for this. I will add a way to specify which version to use.

@Nokel81 Nokel81 added the p1 label Mar 23, 2023
@jseletz-rave
Copy link

+1 - I am seeing double the CPU and Memory requests for containers than what are actually requested.

@7wingfly
Copy link

7wingfly commented May 2, 2023

I've noticed that it starts off with the correct metrics when the nodes are created but then suddenly doubles everything after a few minutes
image

@vilianvenkov
Copy link

+1
Screenshot 2023-05-10 at 18 34 04

@jweak
Copy link
Contributor

jweak commented May 22, 2023

Hey guys, I'm unable to reproduce this issue with https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack installed. Could you guys share a bit more about your setup?

What kube version are you using? Which version of the kube-prometheus-stack are you using? Does this bug happen with all deployments? Are you all having both node and pod metrics doubled?

No need to post anything sensitive but some example would go a long way fixing, thanks!

@jseletz-rave
Copy link

I am running with the "Builtin Metrics Provider" automatically installed by Lens via the cluster settings. K8s 1.23. All deployments. Both Node and pod metrics are doubled.

image

@lghinet
Copy link

lghinet commented May 23, 2023

hello,
I have kube-prometheus-stack-45.26.0 installed on a new AKS 1.26.3 (running
mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 in 2 replicas )

pod memory is doubled

@karkill
Copy link
Author

karkill commented May 23, 2023

Hello bro,
I use kube-prometheus-stack version 45.4.0 (app version v0.63.0) in EKS 1.23.
Errors: CPU, RAM of nodes is doubled.
Lens: 2023.5.120950-latest
I have metrics-server run with 2 replicas but I try scale to 1 but still errors.
image

@AleksuKey
Copy link

Same problem here!

@jweak
Copy link
Contributor

jweak commented May 25, 2023

Alright, I think there are two different issues. One with double pod metrics which is from the referred commit and double node metrics which is something different. We will probably have a setting that let's you change the query described in this PR #7777

The node double metrics is still a bit unclear why this happens. The query has not changed in a while.

@JaySeletz
Copy link

Any progress on this? It's been broken for quite a while?

@gavinclarkeuk
Copy link

Not sure if it is relevant, but I'm seeing an issue where the requests for Succeeded pods (e.g. from Jobs) are being included in the Node level metrics. This means the node level graphs are always overstating the current usage.

@karkill
Copy link
Author

karkill commented Aug 23, 2023

Today, when I checked kube_node_status_capacity in prometheus, I saw another data from kubecost. After uninstalling it, everything was back to normal. I'll close this issuses now. Thank for help.
image

@karkill karkill closed this as completed Aug 23, 2023
@sawaca96
Copy link

sawaca96 commented Sep 1, 2023

Maybe if I use metrics-server and kube-state-metrics together, it will double?

@tonyemiliano
Copy link

I’m using Lens 2023.9.290703-Latest, Kube-Prometheus-stack version: 51.2.0
appVersion: v0.68.0
Im using GKE 1.24.15-gke.1700

I installed in the "monitoring" namespace Kube-Prometheus-stack, I’m having problems only in the visualization of the graphics of the nodes. Doesn’t seem to double the value, doesn’t make much sense to me.

All nodes there have the wrong memory value relative to the graph when we click on the node that is where it seems to be right.

image

I’ve removed everything from namespaces that had other Prometheus installations. Everything seems to work perfect. Minus the memory graphics on the nodes page.

Strange that in my other staging cluster with the same settings is working.

@catemiko
Copy link

I’m using Lens 2023.9.290703-Latest, Kube-Prometheus-stack version: 51.2.0 appVersion: v0.68.0 Im using GKE 1.24.15-gke.1700

I installed in the "monitoring" namespace Kube-Prometheus-stack, I’m having problems only in the visualization of the graphics of the nodes. Doesn’t seem to double the value, doesn’t make much sense to me.

All nodes there have the wrong memory value relative to the graph when we click on the node that is where it seems to be right.

image

I’ve removed everything from namespaces that had other Prometheus installations. Everything seems to work perfect. Minus the memory graphics on the nodes page.

Strange that in my other staging cluster with the same settings is working.

That issue may occur when you have more than one kube-prometheus kubelet in namespace kube-system. You can see all services in kube-system by running kubectl -n kube-system get service -o wide

@hyp3ri0n-ng
Copy link

Anyone know why this issue is marked as closed? I'm still seeing the same on my lens installation version 2024.3.70925-latest) with a desktop pro subscription. In fact it seems when I close and open lens everything has doubled again.... one of my pods is now using 90GB of memory (it's actually using 6-9)...

Would be good to know if there is anything that's being worked on with this issue. If not I'm going to have to cancel that subscription :-/ and I really don't want to. Lens has all the info I need and it's presented REALLY nicely. The big problem is that it just uhh, doesn't work....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/metrics All the things related to metrics bug Something isn't working p1
Projects
None yet
Development

No branches or pull requests