Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
High kmalloc-32 slab cache consumption with 10k containers
Hi, When running 10000 (more-or-less-empty-)containers on a bare-metal Power9 server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory consumption increases quite a lot (around 172G) when the containers are running. Most of it comes from slab (149G) and within slab, the majority of it comes from kmalloc-32 cache (102G) The major allocator of kmalloc-32 slab cache happens to be the list_head allocations of list_lru_one list. These lists are created whenever a FS mount happens. Specially two such lists are registered by alloc_super(), one for dentry and another for inode shrinker list. And these lists are created for all possible NUMA nodes and for all given memcgs (memcg_nr_cache_ids to be particular) If, A = Nr allocation request per mount: 2 (one for dentry and inode list) B = Nr NUMA possible nodes C = memcg_nr_cache_ids D = size of each kmalloc-32 object: 32 bytes, then for every mount, the amount of memory consumed by kmalloc-32 slab cache for list_lru creation is A*B*C*D bytes. Following factors contribute to the excessive allocations: - Lists are created for possible NUMA nodes. - memcg_nr_cache_ids grows in bulk (see memcg_alloc_cache_id() and additional list_lrus are created when it grows. Thus we end up creating list_lru_one list_heads even for those memcgs which are yet to be created. For example, when 10000 memcgs are created, memcg_nr_cache_ids reach a value of 12286. - When a memcg goes offline, the list elements are drained to the parent memcg, but the list_head entry remains. - The lists are destroyed only when the FS is unmounted. So list_heads for non-existing memcgs remain and continue to contribute to the kmalloc-32 allocation. This is presumably done for performance reason as they get reused when new memcgs are created, but they end up consuming slab memory until then. - In case of containers, a few file systems get mounted and are specific to the container namespace and hence to a particular memcg, but we end up creating lists for all the memcgs. As an example, if 7 FS mounts are done for every container and when 10k containers are created, we end up creating 2*7*12286 list_lru_one lists for each NUMA node. It appears that no elements will get added to other than 2*7=14 of them in the case of containers. One straight forward way to prevent this excessive list_lru_one allocations is to limit the list_lru_one creation only to the relevant memcg. However I don't see an easy way to figure out that relevant memcg from FS mount path (alloc_super()) As an alternative approach, I have this below hack that does lazy list_lru creation. The memcg-specific list is created and initialized only when there is a request to add an element to that particular list. Though I am not sure about the full impact of this change on the owners of the lists and also the performance impact of this, the overall savings look good. Used memory Before During After W/o patch 23G 172G 40G W/ patch 23G 69G 29G Slab consumption Before During After W/o patch 1.5G 149G 22G W/ patch 1.5G 45G 10G Number of kmalloc-32 allocations Before During After W/o patch 178176 3442409472 388933632 W/ patch 190464 468992 468992 Any thoughts on other approaches to address this scenario and any specific comments about the approach that I have taken is appreciated. Meanwhile the patch looks like below: From 9444a0c6734c2853057b1f486f85da2c409fdc84 Mon Sep 17 00:00:00 2001 From: Bharata B Rao <bharata@linux.ibm.com> Date: Wed, 31 Mar 2021 18:21:45 +0530 Subject: [PATCH 1/1] mm: list_lru: Allocate list_lru_one only when required. Don't pre-allocate list_lru_one list heads for all memcg_cache_ids. Instead allocate and initialize it only when required. Signed-off-by: Bharata B Rao <bharata@linux.ibm.com>
- Loading branch information