From 07c14672bfe71caccc6a03bbb4b584a3d783dd6b Mon Sep 17 00:00:00 2001 From: Oran Agra Date: Sun, 18 Jun 2023 08:30:23 +0100 Subject: [PATCH] Set Jemalloc --disable-cache-oblivious to reduce memory overhead (#12315) Apparently for large size classes Jemalloc allocate some extra memory (can be up to 25% overhead for allocations of 16kb). see https://github.com/jemalloc/jemalloc/issues/1098#issuecomment-1589870476 p.s. from Redis's perspective that looks like external fragmentation, (i.e. allocated bytes will be low, and active pages bytes will be large) which can cause active-defrag to eat CPU cycles in vain. Some details about this mechanism we disable: --------------------------------------------------------------- Disabling this mechanism only affects large allocations (above 16kb) Not only that it isn't expected to cause any performance regressions, it's actually recommended, unless you have a specific workload pattern and hardware that benefit from this feature -- by default it's enabled and adds address randomization to all large buffers, by over allocating 1 page per large size class, and offsetting into that page to make the starting address of the user buffer randomized. Workloads such as scientific computation often handle multiple big matrixes at the same time, and the randomization makes sure that the cacheline level accesses don't suffer bad conflicts (when they all start from page-aligned addresses). However the downsize is also quite noticeable, like you observed that extra page per large size can cause memory overhead, plus the extra TLB entry. The other factor is, hardware in the last few years started doing the randomization at the hardware level, i.e. the address to cacheline mapping isn't a direct mapping anymore. So there's debate to disable the randomization by default, but we are still hesitant because when it matters, it could matter a lot, and having it enabled by default limits that worst case behavior, even though it means the majority of workloads suffers a regression. So in short, it's safe and offers better performance in most cases. --- deps/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/Makefile b/deps/Makefile index 34656201d0d2..c03c79790cb8 100644 --- a/deps/Makefile +++ b/deps/Makefile @@ -106,7 +106,7 @@ endif jemalloc: .make-prerequisites @printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR) - cd jemalloc && ./configure --disable-cxx --with-version=5.3.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" $(JEMALLOC_CONFIGURE_OPTS) + cd jemalloc && ./configure --disable-cxx --with-version=5.3.0-0-g0 --with-lg-quantum=3 --disable-cache-oblivious --with-jemalloc-prefix=je_ CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" $(JEMALLOC_CONFIGURE_OPTS) cd jemalloc && $(MAKE) lib/libjemalloc.a .PHONY: jemalloc