Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
client-go - ThreadSafeMap indicies - empty sets not deleted - memory increase with high cardinality short lived resources #84959
Consider this use case, a k8s cluster with many short lived name spaces with a high cardinality. (Think UUIDs).
Over time, some of the
Leads to process memory increasing over time until the process exceeds available memory. Potentially causing issues as an api-server or controller is terminated by the operating system.
What you expected to happen:
Process memory does not increase unbounded as the number of unique items added then deleted from a thread safe map, causing indices to hold on to zero length string sets for the key.
This could be mitigated when an index's string set for a key is empty, that string set is removed from the index.
How to reproduce it (as minimally and precisely as possible):
Thanks to @awprice for this reproduction code:
Anything else we need to know?:
Can be mitigated with the following patch against master at 75aca1f:
Here's an apiserver pprof heap base png from a clean start for a test case of ≈800k name space create/delete in kube v1.14.7:
Note the increase in heap utilisation from base of
Same as above, but with the suggested fix and ≈100k name space create/delete:
Note lack of
(Was not able to directly attach the images to this issue due to Something went really wrong, and we can't process that file.)
@tedyu I'm not sure I follow, but
One option is to mark the time when set becomes empty:
We can clean Index entry when certain period has passed since the set becomes empty.
@tedyu I really don't think that solution is a viable and adds unneeded complexity. You would then need to have something running in the background to prune empty sets.
The solution proposed in the original post is the most straight forward, when the set is empty - delete it. If something is then created that needs the set, it will be recreated when needed.