High CPU usage (32 vCPUs) - looks due to targets discovery in K8s #8014
Comments
|
Removing a few jobs that weren't scraping any targets (~20) did actually reduce the CPU load significantly (~3 vCPU) |
|
Can we have the startup logs (if possible loglevel debug)? |
|
Here's the log from the past few days, including startup (I'm seeing a lot of K8s API errors, which I believe are related to transient instability of that managed cluster). This is a fresh start with debug level. |
|
You are opening 28 different kubernetes connections. You could try to reuse the exact same service discoveries config and apply relabel_configs per job to select which targets to monitor. cat prod1c-prom-logs-debug.txt |grep subs |grep kuber -c |
|
Thanks for the logs anyway, I will try to have a look at the code as well. |
|
@roidelapluie thanks for the suggestion; can you point me to an example of such config? I then need to check if that's possible via PromOp ServiceMonitors or something similar... |
|
Basically if the bit under |
|
So basically I need to get rid of the namespace in the I believe that's not (currently) possible with ServiceMonitors, not even with a more general I'll ask around and possibly reach out to PromOp to see if there are ways to do that. However having the namespace filter in the |
|
What's more efficient would depend on how many namespaces there are, and what proportion of them that this Prometheus is interested in. I'd expect specifying a namespace to always be more efficient though, so if this is the issue this smells like a performance issue in the k8s client code somewhere as the namespace filtering isn't happening properly on the server side. |
|
@brian-brazil we have and |
|
Any insights here would be helpful. we are getting to 500 nodes with > 2000 pods and right now we would need to get > 32 CPU for prometheus. |
There is no namespace level filtering on the server-side that I'm aware of. There's two ways you can implement it. 1 'watch all namespaces' watcher, with client side filtering, or open 1 watch per-namespace (with http2 that can still be 1 connection if you re-use the client I believe). |
|
So we managed to merge together a lot of ServiceMonitors. Even though they were all scraping the same namespace, we got an impressive reduction in CPU usage (less than half; see below). I think this indicates the number of connections to Kubernetes (which should be one per namespace in this scenario, as suggested previously) might not be the primary contributor, while the sheer number of jobs that use K8s Service Discovery is. Just to reiterate on this: is my assumption the service discovery for each target pool runs every 5 seconds correct (see here)?
|
No, that's a throttle so we don't process updates from an SD more than every 5s. Processing updates is considered to be cheap, but not that cheap. |
|
We are currently facing the same issue on our Prometheis instances: we have ~30 kube sd jobs (pod type) that are not constrained by namespaces and there are a lot of pods running on the platform (~5K pods) and the CPU usage for those instances is abnormally high. NB: we also noticed that the rate on prometheus_target_sync_length_seconds_sum dropped after adding the selectors. This is a sum but if you get the details for all the scrape jobs, the jobs with the new selector drop to ~0 and the other jobs see their rate dropping. |
|
Can you get a CPU profile? |
|
CPU profile with the selectors CPU profile without selectors |
|
Looks like it's that part of the code alright. It's not something we've ever really optimised, as it's not meant to be run particularly often. k8s sending 28k targets for each of 30 scrape configs every 5 seconds wasn't exactly in mind. There's probably some low hanging fruit here. |
|
@brian-brazil Hey there! What if we have a label filter in the k8s discovery plugin as a fast win? In our case discovery produces ~120 labels per target, but we use only 5 of them. Yes, there is a relabel config, but the main performance issue is to alloc memory for the number of labels (30000 targets * 120 labels=~3 million labels), sort them, hash for deduplication, and then send to relabeling. This long and hard work happens each scrape pool sync. What do you think? |
|
That sounds like slowly reinventing relabelling, so would ultimately end up with the same performance costs. We have to allocate the labels one way or the other. I'd suggest looking at ways to make what we have more efficient, rather than immediately jumping to adding yet more configuration for users. There's likely quite a few low hanging fruit performance wise, as the relevant code paths have never really been optimised. |
|
The option from @vitkovskii sounds really reasonable. Unfortunately, we just wasting CPU and memory for useless work here and I have no idea what to do next because the number of targets in our K8S growing really rapidly. It seems that Prometheus becoming unscalable here. Let's find an engineering solution, colleagues. |
|
Actually, this hard work is doing per service monitor. So if you have 30 of them, you have x30 useless work. Let's count: 120 labels is about 6Kb, 30000 targets, and 30 service monitors. It's about 5.5Gb of RAM only to create raw label sets from discovery. |
|
Do you have concrete exemples about how we could avoid that? Is that e.g. underlying labels you could delete from your pods? |
That'd only be if none of them were dropped, which is unlikely to be the case. We should only be keeping dropped targets once, not 30x - if not that sounds like a low hanging fruit that could be tackled. |
It's definitely amplified by the number of Service Monitors. We replaced tens of SM with a single one (for a particular application), while the overall pods/labels stayed the same (or even slightly increased) and the CPU usage got to less than 50% it was before (see #8014 (comment)) |
I'm confused here. #8014 (comment) is talking about RAM, and now you're talking about CPU. Which is the problem? |
|
For us it was definitely CPU (as per the title of the issue). Not sure about RAM, haven't seen any correlation with a change in memory usage yet. |
|
Let's just stick with CPU then. If it is also memory that's a separate thing which we can look at optimising. My first though on how to handle this generally so it benefits all SDs/users would be to see if we can avoid re-process a target that hasn't changed when there's refresh from SD, thus avoiding a large chunk of the relabelling processing. |
|
Yeah, looking at the code at https://github.com/prometheus/prometheus/blob/master/scrape/scrape.go#L417-L431 if we made droppedTargets a map we could check if an identical target was dropped last time and fastpath that, bypassing all the relabelling. |
We could sort them in the UI, if that is really making a big difference. But I think that relabeling might somwhow have dependencies on order, alonside other sanity checks we do (like checking for duplicate labels) |
Yes, that's part of our documented API that cannot be broken - though the sorting doesn't matter, it is a JSON map which has no ordering.
For the data structure to work it needs to be sorted, and that includes for relabelling so it can find the labels. Are we sorting more than we need to? We should only need to do it once per SD target. |
|
The first one is here: https://github.com/prometheus/prometheus/blob/master/scrape/target.go#L368 actually, it's not necessary for relabeling. The second time after relabeling: https://github.com/prometheus/prometheus/blob/master/scrape/target.go#L425. The second one is OK because next we should calculate |
Relabeling scans all labels and it's not required them to be sorted. |
|
Why relabeling take a list of labels, not a map? For every target on every config iteration, it allocates a new list. This is the second bottleneck after sorting. |
|
The map is the old data structure, Prometheus 2.x introduced the list for performance reasons. The map should be considered legacy, and remaining uses removed where practical. Why not do the sort up in SD? We only need it once, not for every scrape config and it'd further prune where the old data structure is used.
It's using a library which expects them to be sorted, so that invariant should be maintained. Relabelling does require them to be sorted due to this. |
|
@brian-brazil there are any plans to improve that area? it's still a pain for anybody who uses many service monitors in his cluster, the savings potential here is great. I'll donate my CPU profiling here, 172 service monitors, 5 cores currently in use, we can see that at least half of them being wasted for the same
|
|
I think that's the first thing we should probably look at improving. |
|
@shaikatz do you mind uploading the whole profile either as a file to github or via https://share.polarsignals.com/ ? It would be great if we could explore the profile. I also opened prometheus-operator/prometheus-operator#3840 as I think that is another angle from which this can be optimized on the prometheus-operator side. |
|
Could you also share an allocs profile? From that profile it does look to me like a lot of memory-trash is produced by discovery which causes large sweep and GC CPU usage. It does all seem to add up to prometheus-operator/prometheus-operator#3840. |
|
Is that the one you need? profile001.svg.zip |
|
Yeah it is, thank you! Yeah that seems to also point in the same direction. |
|
@brancz / @brian-brazil / @roidelapluie is there any update on this issue? any options we can explore to patch this behavior? |
|
Same problem here. |
|
As MR was not accepted we've decided to implement our own separate discovery service (we're calling it "target balancer") and deliver targets in files (fileSD) to Proms. Profits: 1) reduce kubeAPI load 2) significantly reduced CPU usage on Prom instances |
|
It seems we are also hitting this issue, but the weird thing is we are hitting the issue with only 1 of our prometheus servers!! (we have configured 2 replicas in prometheus operator, no sharding). However the cpu usage is mostly in prometheus-0: profile.pb.gz prometheus-1: profile(1).pb.gz You can see the difference in pprof between two instances. Also the difference in CPU usage of these instances:
More info: |
|
@m-yosefpor we finally moved out all discovery logic outside of prometheus to separate daemon and switched to simple file/http discovery, it also allowed us to implement custom sharding logic. |
We would appreciate if such tools gets open-sourced. |
|
Need a great solution, currently we set |
I can't figure out why this could happen. What library is being used? |
What did you do?
We're monitoring a Kubernetes cluster consisting of about 400 nodes and 4500 pods, leveraging a single Prometheus instance with 32 vCPUs (almost fully utilized, while memory hovers between 40-50Gi). The setup leverages Prometheus Operator and most of the targets come from Service Monitors definitions (shouldn't be too relevant for the issue though).
There are about 130 target pools, with a few of those that result each in a few hundreds pods to scrape (a handful can have a couple thousands pods).
Judging by the CPU profiling graph, it looks like most of the CPU is used to update those target pools.
pprof.prometheus.samples.cpu.005.pb.gz
EDIT: We're experiencing the same in another cluster with much less total Pods (~1500) but way higher target pools (~450).
What did you expect to see?
Not exactly sure what overall CPU usage to expect for such load, but definitely not >60% of 32vCPUs for targets discovery alone.
In case this usage is expected (and provided it is indeed coming from targets discovery), I would expect to be able to set a custom interval for targets update to tune such behavior or some other ways to reduce CPU footprint.
What did you see instead? Under which circumstances?
32 vCPUs (almost fully utilized), >60% of which seems to be related to targets discovery.
I see about 80 of such pools taking more than 5 seconds to get synched (varying between 4 and 8 seconds).
If my understanding is correct, the sync is executed every 5 seconds (
prometheus/scrape/manager.go
Line 158 in bd53b5f
Environment
Linux 4.15.0-1093-azure x86_64
prometheus, version 2.20.1 (branch: HEAD, revision: 983ebb4)
build user: root@7cbd4d1c15e0
build date: 20200805-17:26:58
go version: go1.14.6
PS: I can provide the full configuration if that's helpful, though it's quite longer
The text was updated successfully, but these errors were encountered: