Kernel consumption too high

[jochen-schuettler]

What happened?

The measured consumption for the kernel is much too high, beyond what the machine is physically capable of.

What did you expect to happen?

Correct measurement for kernel.

How can we reproduce it (as minimally and precisely as possible)?

Install kepler 0.7.7 into AWS ECS. Observe measured kernel consumption over some days.

Anything else we need to know?

No response

Kepler image tag

0.7.7

Kubernetes version

1.24

Cloud provider or bare metal

AWS EKS / ECS

OS version

Amazon Linux 2

Install tools

Helm

Kepler deployment config

Unchanged from Helm chart.

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

CNI Version: v1.12.0-eksbuild.2 CSI Version: v1.14.0-eksbuild.1

[rootfs]

I suspect this is a overflow issue. Do you have this issue before or only during this period?

[jochen-schuettler]

it started with 0.7.3 -> 0.7.7, while 0.7.3 had other issues. ongoing problem

[jochen-schuettler]

namespace "tools" is also affected

[jochen-schuettler]

compare namespace "kepler" in the same frame

[marceloamaral]

@jochen-schuettler thanks for investigating this.

• Are you running on bare-metal or VM?
• Can you check the metrics in prometheus:
  • rate(kepler_container_bpf_cpu_time_ms_total[1m])
  • rate(kepler_container_task_clock_ms_total[1m])
  • rate(kepler_container_package_joules_total[1m])
  • rate(kepler_container_dram_joules_total[1m])
  • rate(kepler_container_other_joules_total[1m])
  • rate(kepler_node_package_joules_total[1m])
  • rate(kepler_node_dram_joules_total[1m])
  • rate(kepler_node_other_joules_total[1m])

[jochen-schuettler]

It's a VM on AWS ECS. I don't have access to Prometheus, but to Grafana. This is the first metric, rate(kepler_container_bpf_cpu_time_ms_total[1m])

[jochen-schuettler]

rate(kepler_container_task_clock_ms_total[1m])

[jochen-schuettler]

rate(kepler_container_package_joules_total[1m])

[jochen-schuettler]

rate(kepler_container_dram_joules_total[1m])

[jochen-schuettler]

rate(kepler_container_other_joules_total[1m])

[jochen-schuettler]

rate(kepler_node_package_joules_total[1m])

[jochen-schuettler]

rate(kepler_node_dram_joules_total[1m])

[jochen-schuettler]

rate(kepler_node_other_joules_total[1m])

[marceloamaral]

@sunya-ch the power models are using bpf_cpu_time right?
Any idea about the huge numbers?

[jochen-schuettler]

(no change by v0.7.8)

[jochen-schuettler]

@sunya-ch : see above, any idea ?

[jochen-schuettler]

The issue is still open. @marceloamaral , @rootfs : Can I help you in any way to investigate ?

[TWpgo]

I also get some really unrealistic measurements from time to time

[jochen-schuettler]

Additional Info: The VM is running on a AMD EPYC 7R13 Processor (stepping 1). This is not included in cpus.yaml. So maybe the problem is related to that?
Edit: After further Reading I realized that Milan, Zen 3 is eq. EPYC 7R13, so the CPU is in the list after all.
1.) But does Kepler know that?
2.) Can we somehow access which CPU model was used?
3.) Also, is it possible these peaks happen when the VM host changes? Maybe AWS EKS shifts a VM quite freely between physical hosts, leading to steps in CPU time? (This is a total guess from an outsider's point of view.)

[rootfs]

Maybe there are some overflow during extended tests? @vprashar2929 have you seen this in any tests?

[rootfs]

@jochen-schuettler can you get prometheus stats kepler_container_joules_total over the period of spikes? This stats is used to calculate the power consumption.

[jochen-schuettler]

[jochen-schuettler]

Total Power Consumption (W) in Namespace All-data-2024-04-10 09 15 31.csv

[vprashar2929]

@rootfs I have seen this before and in the case of VM's Kepler does report higher energy usage. I think post 0.7.2 only it started. #1142

[jochen-schuettler]

We have occurences regardless of the kernel namespace in a local setup in both BM and VM. Looking at bpf_cpu_time_ms there reveals rapid rises over some minutes. kepler 0.7.2 seems ok, yes.