From 0575860d064694d4e2f307b2c20a880a6a7b59ab Mon Sep 17 00:00:00 2001 From: David Hildenbrand Date: Tue, 12 Apr 2022 10:59:53 +0200 Subject: [PATCH] mm: take most of the "maybe" out of folio_maybe_dma_pinned() for anonymous pages Let's optimize folio_maybe_dma_pinned() for order-0 pages where we mangle the pincount into the refcount. We only allow pinning anonymous pages that are marked exclusive and disallow clearing the exclusive marker while they are pinned. Consequently, there is no valid scenario where a shared anonymous page could be pinned. With this change, shared anonymous order-0 pages will never get detected as "maybe pinned", not even when concurrent GUP-fast temporarily marks them pinned. After this change, we might only get false positives for anonymous pages if: (1) Concurrent GUP-fast temporarily increased the pin counter, before decreasing it again: applies to exclusive anonymous order-0 pages and compound anonymous pages. Rare. (2) We have more than 1024 references on an exclusive order-0 page. While possible in theory, this should be highly unlikely. What could go wrong? Not much. Assuming we'd have pinned a shared anonymous page (bug), folio_maybe_dma_pinned() would now return "false". However, page_try_dup_anon_rmap() and page_try_share_anon_rmap() essentially do nothing if the exclusive marker is not set already. We'd already have a potential memory corruption. So this change primarily only affects mm/vmscan:shrink_page_list() right now, whereby we can now swapout pages that have more than 1024 references: for example, simply when many processes share an order-0 page. Note that once an anonymous page gets unmapped to eventually get freed, the page remains anonymous until actually freed: when the last reference is gone. Similarly, PageAnonExclusive() is only cleared when actually freeing the page. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 878a516f2267bb..d6cd2febad4122 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1619,6 +1619,17 @@ static inline bool folio_maybe_dma_pinned(struct folio *folio) if (folio_test_large(folio)) return atomic_read(folio_pincount_ptr(folio)) > 0; + /* + * We only allow pinning anonymous pages (see gup_must_unshare()) + * while they have the exclusive marker set and don't allow clearing the + * exclusive marker while they maybe pinned (see + * page_try_share_anon_rmap() and page_try_dup_anon_rmap()). + * Consequently, pages with the exclusive marker cleared cannot be + * pinned and we can optimize for that case here. + */ + if (folio_test_anon(folio) && !PageAnonExclusive(&folio->page)) + return false; + /* * folio_ref_count() is signed. If that refcount overflows, then * folio_ref_count() returns a negative value, and callers will avoid