[Deepin-Kernel-SIG] [linux 6.6.y] [Upstream] mm/list_lru: Split list_lru lock into per-cgroup scope

[opsiff]

Commits(8):
mm: zswap: fix objcg use-after-free in entry destruction
mm/list_lru: simplify reparenting and initial allocation
mm/list_lru: code clean up for reparenting
mm/list_lru: don't export list_lru_add
mm/list_lru: don't pass unnecessary key parameters
mm: list_lru: remove unused macro list_lru_init_key()
mm: workingset: dynamically allocate the mm-shadow shrinker
zswap: make shrinking memcg-aware

Link: https://lore.kernel.org/all/20241104175257.60853-1-ryncsn@gmail.com/T/#u
From: Kairui Song kasong@tencent.com

Currently, every list_lru has a per-node lock that protects adding,
deletion, isolation, and reparenting of all list_lru_one instances
belonging to this list_lru on this node. This lock contention is heavy
when multiple cgroups modify the same list_lru.

This can be alleviated by splitting the lock into per-cgroup scope.

To achieve this, this series reworked and optimized the reparenting
process step by step, making it possible to have a stable list_lru_one,
and making it possible to pin the list_lru_one. Then split the lock
into per-cgroup scope.

The result is ~15% performance gain for simple multi-cgroup tar test
of small files, and reduced LOC. See PATCH 5/6 for test details.

V2: https://lore.kernel.org/linux-mm/20240925171020.32142-1-ryncsn@gmail.com/
Updates from V2:

• Collect Acked-by.
• Fix a WARN_ON issue caused by potential compiler optimization issue.
  [Dan Carpenter, Naresh Kamboju]
  https://lore.kernel.org/linux-mm/62a65418-2393-40ec-b462-151605a5efcf@stanley.mountain/
  [Applied to "mm/list_lru: split the lock to per-cgroup scope"]
• Fix a BUG_ON issue, V2 forgot to cover user of LRU_STOP. [Usama Arif]
  https://lore.kernel.org/linux-mm/CAMgjq7D_OA=vYf5SnNnKXjppPFhDqsbYF--6=cOayKiadxuwrQ@mail.gmail.com/

V1: https://lore.kernel.org/linux-mm/20240624175313.47329-1-ryncsn@gmail.com/
Updates from V1:

• Collect Review-by.
• Fix a race of initialization issue that may lead to mem leak [Muchun
  Song]
• Drop a unrelated and incorrect fix [Shakeel Butt]
• Use VM_WARN_ON instead of WARN_ON for several sanity checks.

Kairui Song (6):
mm/list_lru: don't pass unnecessary key parameters
mm/list_lru: don't export list_lru_add
mm/list_lru: code clean up for reparenting
mm/list_lru: simplify reparenting and initial allocation
mm/list_lru: split the lock to per-cgroup scope
mm/list_lru: simplify the list_lru walk callback function

drivers/android/binder_alloc.c | 8 +-
drivers/android/binder_alloc.h | 2 +-
fs/dcache.c | 4 +-
fs/gfs2/quota.c | 2 +-
fs/inode.c | 5 +-
fs/nfs/nfs42xattr.c | 4 +-
fs/nfsd/filecache.c | 5 +-
fs/xfs/xfs_buf.c | 2 -
fs/xfs/xfs_qm.c | 6 +-
include/linux/list_lru.h | 26 ++-
mm/list_lru.c | 383 ++++++++++++++++-----------------
mm/memcontrol.c | 10 +-
mm/workingset.c | 20 +-
mm/zswap.c | 12 +-
14 files changed, 241 insertions(+), 248 deletions(-)

--
2.47.0

Summary by Sourcery

Integrate memcg-aware list_lru support into zswap and core MM infrastructure to enable cgroup-aware zswap reclaim and robust handling of memcg offlining and swap cache recursion.

New Features:

• Add memcg-aware list_lru integration for zswap entries to support per-memcg LRU accounting and reclaim.
• Introduce cgroup-aware zswap reclaim that walks memcg-specific LRUs and selects memcgs in a round-robin fashion for global reclaim.
• Provide zswap_memcg_offline_cleanup to keep pool state consistent during memcg offlining and reparenting.

Bug Fixes:

• Prevent potential deadlock/recursion in __read_swap_cache_async by allowing callers to skip waiting when the swap cache entry already exists but is not yet attached.
• Avoid reclaiming from offlined (zombie) memory cgroups during zswap reclaim to prevent stale references and incorrect accounting.

Enhancements:

• Convert zswap’s per-pool LRU from a simple list+lock to the generic list_lru infrastructure with memcg awareness.
• Charge and allocate zswap entries on the appropriate NUMA node and attach them to the correct memcg-aware list_lru instance.
• Rework list_lru’s memcg integration, including reparenting and allocation paths, to use list_lru_one and Xarray-based tracking more safely under concurrent operations.
• Update workingset shadow node shrinker initialization to use the new list_lru memcg+lockdep helpers and dynamic shrinker allocation/registration.
• Refine memcg offlining paths to reparent list_lru data structures safely and invoke zswap-specific cleanup before kmem and shrinker reparenting.

[sourcery-ai]

Reviewer's Guide

Make zswap reclaim and list_lru fully memcg-aware, switching zswap pool LRU management from a manual list + spinlock to the generic list_lru infrastructure, updating memcg list_lru reparenting/allocation, and fixing a swap cache recursion hazard during zswap-triggered reclaim.

Sequence diagram for memcg-aware zswap reclaim and swap cache recursion protection

sequenceDiagram
    participant Task as task
    participant ZS as zswap_store
    participant SM as shrink_memcg
    participant LL as list_lru_walk_one
    participant CB as shrink_memcg_cb
    participant WB as zswap_writeback_entry
    participant SC as __read_swap_cache_async

    Task->>ZS: zswap_store(folio)
    ZS->>ZS: get_obj_cgroup_from_folio(folio)
    ZS->>ZS: obj_cgroup_may_zswap(objcg)?
    alt objcg not allowed
        ZS->>ZS: get_mem_cgroup_from_objcg(objcg)
        ZS->>SM: shrink_memcg(memcg)
        SM->>SM: zswap_pool_current_get()
        SM->>LL: list_lru_walk_one(pool->list_lru, nid, memcg, shrink_memcg_cb)
        LL-->>CB: shrink_memcg_cb(item, lru_one, lock, arg)
        CB->>WB: zswap_writeback_entry(entry, tree)
        WB->>SC: __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true)
        SC-->>WB: page or error
        WB-->>CB: writeback_result
        CB-->>LL: LRU_REMOVED_RETRY or LRU_RETRY
        SM-->>ZS: ret (0 or error)
    end
    ZS-->>Task: store result / reject

Loading

File-Level Changes

Change	Details	Files
Refactor zswap entry LRU management to use memcg-aware list_lru instead of a per-pool list_head + spinlock, and make reclaim operate per memcg with proper RCU and node-awareness.	Include linux/list_lru.h and replace zswap_pool lru/list + spinlock fields with struct list_lru and a mem_cgroup *next_shrink cursor. Add helper functions to derive mem_cgroup and NUMA node from zswap entries, and implement zswap_lru_add/zswap_lru_del/zswap_lru_putback wrappers using list_lru APIs under RCU. Introduce zswap_memcg_offline_cleanup to advance next_shrink when a memcg is offlined and hook it into mem_cgroup_css_offline. Change zswap_entry allocation to be NUMA-aware via kmem_cache_alloc_node. Ensure zswap_free_entry/uncharge paths use list_lru_del and update ordering of objcg uncharge/put. Replace zswap_reclaim_entry with shrink_memcg_cb + shrink_memcg + round-robin shrink_worker that iterates memcgs, uses mem_cgroup_iter and mem_cgroup_tryget_online, and walks list_lru per-node via list_lru_walk_one. Initialize and destroy zswap_pool list_lru using list_lru_init_memcg/list_lru_destroy, and clean up next_shrink/mem_cgroup_iter on pool destroy. Update zswap_store/zswap_load to allocate memcg list_lru state (memcg_list_lru_alloc), add entries to list_lru, and perform targeted shrink_memcg when obj_cgroup_may_zswap() rejects the store.	mm/zswap.c
Evolve list_lru memcg support to work on mem_cgroup pointers instead of raw kmemcg_id indices, improve reparenting semantics, and simplify allocation of list_lru_memcg structures.	Add list_lru_from_memcg helper (CONFIG_MEMCG_KMEM and non-MEMCG variants) that resolves the appropriate list_lru_one for a memcg, reparenting up the hierarchy when necessary. Change list_lru_add/list_lru_del to call list_lru_from_memcg instead of list_lru_from_memcg_idx, and drop EXPORT_SYMBOL_GPL(list_lru_add). Remove memcg_list_lru_free and the old memcg_reparent_list_lru/memcg_reparent_list_lrus implementation that relied on kmemcg_id updates and post-reparent freeing of per-memcg list_lrus. Introduce a new memcg_reparent_list_lrus that iterates memcg_list_lrus, clears the xarray entry for the memcg, reparenting per-node lists under lru->node[i].lock and freeing the list_lru_memcg via kvfree_rcu. Rewrite memcg_list_lru_alloc to allocate and publish list_lru_memcg structures incrementally for the farthest ancestor without list_lru state, using xarray locking, css_is_dying checks, and simplified error/nomem handling. Adjust __list_lru_init lockdep integration to use a key stored in struct list_lru (under CONFIG_LOCKDEP) instead of passing a key into the initializer.	mm/list_lru.c include/linux/list_lru.h
Rework workingset shadow node shrinker initialization to use the new shrinker/list_lru APIs and lockdep key storage.	Remove the static workingset_shadow_shrinker struct and legacy prealloc_shrinker/register_shrinker_prepared usage. In workingset_init, allocate a shrinker via shrinker_alloc with NUMA+MEMCG flags, then initialize the shadow_nodes list_lru using list_lru_init_memcg_key with a lock_class_key stored on the list_lru. Set shrinker callbacks (count_shadow_nodes, scan_shadow_nodes), seeks=0, and register via shrinker_register; on error, teardown with shrinker_free.	mm/workingset.c include/linux/list_lru.h
Extend swap cache async read to guard against recursion when zswap-triggered reclaim calls back into __read_swap_cache_async for the same entry.	Add a skip_if_exists boolean parameter to __read_swap_cache_async and to its declaration in mm/swap.h. Within __read_swap_cache_async, after a failed add_to_swap_cache with -EEXIST, bail out early if skip_if_exists is true to avoid waiting for a folio that is not yet in swap cache. Update all existing call sites (read_swap_cache_async, swap_cluster_readahead, swap_vma_readahead) to pass skip_if_exists=false, and change zswap_writeback_entry to pass skip_if_exists=true when reading swap cache for writeback.	mm/swap_state.c mm/swap.h mm/zswap.c
Hook zswap into memcg offlining and provide a stub for non-memcg builds so the new cleanup call is always available.	Declare zswap_memcg_offline_cleanup in include/linux/zswap.h with a no-op inline stub for !CONFIG_ZSWAP. Implement zswap_memcg_offline_cleanup in mm/zswap.c to advance pool->next_shrink when the current memcg is offlined, under zswap_pools_lock. Call zswap_memcg_offline_cleanup from mem_cgroup_css_offline before memcg_offline_kmem/reparenting to keep zswap’s memcg iteration state consistent.	include/linux/zswap.h mm/zswap.c mm/memcontrol.c
Add a helper for deriving mem_cgroup from obj_cgroup in memcontrol headers, with a stub in the non-memcg configuration.	Declare get_mem_cgroup_from_objcg in include/linux/memcontrol.h, returning NULL in the non-memcg case, for use in zswap_store when performing memcg-aware reclaim decisions.	include/linux/memcontrol.h

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[deepin-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from opsiff. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• deepin/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sourcery-ai]

Hey - I've left some high level feedback:

• In the !CONFIG_MEMCG_KMEM path, the new list_lru_from_memcg() helper still takes an int parameter but is now called with a struct mem_cgroup * in list_lru_add/list_lru_del, which will cause unsafe pointer-to-int coercions; consider aligning the prototype and call sites to avoid type mismatches in non-memcg-kmem builds.
• The new shrink_memcg() path is invoked from zswap_store() when obj_cgroup_may_zswap() fails; it would be useful to double-check that this reclaim-on-store behavior cannot lead to unexpected reentrancy or long latencies in hot paths, e.g. by guarding against nested zswap reclaim calls or imposing tighter bounds on the reclaim work done synchronously.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In the !CONFIG_MEMCG_KMEM path, the new list_lru_from_memcg() helper still takes an int parameter but is now called with a struct mem_cgroup * in list_lru_add/list_lru_del, which will cause unsafe pointer-to-int coercions; consider aligning the prototype and call sites to avoid type mismatches in non-memcg-kmem builds.
- The new shrink_memcg() path is invoked from zswap_store() when obj_cgroup_may_zswap() fails; it would be useful to double-check that this reclaim-on-store behavior cannot lead to unexpected reentrancy or long latencies in hot paths, e.g. by guarding against nested zswap reclaim calls or imposing tighter bounds on the reclaim work done synchronously.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[Copilot]

Pull request overview

This PR backports mainline MM changes to make zswap’s reclaim and accounting memcg-aware by integrating zswap entries with the generic list_lru infrastructure, while also updating core list_lru memcg/reparenting logic and tightening swap-cache behavior to avoid recursion hazards.

Changes:

• Convert zswap’s per-pool LRU to list_lru with memcg-aware accounting and implement memcg round-robin reclaim.
• Rework list_lru memcg integration/reparenting and update workingset shadow shrinker initialization to the newer shrinker/list_lru helpers.
• Extend __read_swap_cache_async() with a “skip if exists” mode to prevent recursive wait/lock scenarios during zswap-triggered reclaim.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
mm/zswap.c	Switches zswap LRU to list_lru, adds memcg-aware reclaim and memcg offline cleanup integration.
mm/workingset.c	Migrates workingset shadow shrinker init to dynamically allocated shrinker + new list_lru init helper.
mm/swap.h	Extends __read_swap_cache_async() signature to add skip_if_exists.
mm/swap_state.c	Implements the skip_if_exists behavior to avoid recursive wait/lock behavior.
mm/memcontrol.c	Hooks zswap memcg offline cleanup into memcg offline sequence.
mm/list_lru.c	Refactors list_lru memcg lookup/reparenting/allocation and lockdep plumbing.
include/linux/zswap.h	Exposes zswap memcg offline cleanup API.
include/linux/memcontrol.h	Provides get_mem_cgroup_from_objcg() (and stub when !MEMCG).
include/linux/list_lru.h	Adjusts list_lru init APIs and adds memcg+lockdep key helper.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[opsiff]

done

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

include/linux/zswap.h:18

• include/linux/zswap.h now declares zswap_memcg_offline_cleanup(struct mem_cgroup *), but this header doesn’t declare/forward-declare struct mem_cgroup or include <linux/memcontrol.h>. This breaks compilation for includers like fs/proc/meminfo.c and mm/page_io.c that include <linux/zswap.h> without pulling in memcontrol definitions (unknown type name ‘struct mem_cgroup’). Add a forward declaration to keep the header self-contained.

void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);

mm/zswap.c:394

• zswap_lru_putback() calls list_lru_putback() after dropping the list_lru walk lock. However, list_lru_putback() currently uses list_lru_from_memcg_idx() and does not fall back to a parent memcg when the per-memcg list has been erased during memcg offlining/reparenting (xa_load can return NULL). That can race with memcg offline (memcg_reparent_list_lrus() erases the xarray entry), leading to a NULL dereference when trying to put an isolated entry back. Consider updating list_lru_putback() (mm/list_lru.c) to use the new list_lru_from_memcg() helper (parent fallback) or otherwise guarantee a non-NULL list during offlining.

	rcu_read_lock();
	memcg = mem_cgroup_from_entry(entry);
	spin_lock(lock);
	/* we cannot use list_lru_add here, because it increments node's lru count */
	list_lru_putback(list_lru, &entry->lru, nid, memcg);