@interwq interwq released this May 8, 2018 · 67 commits to dev since this release

Assets 3

This release is primarily about fine-tuning, ranging from several new features to numerous notable performance and portability enhancements. The release and prior dev versions have been running in multiple large scale applications for months, and the cumulative improvements are substantial in many cases.

Given the long and successful production runs, this release is likely a good candidate for applications to upgrade, from both jemalloc 5.0 and before. For performance-critical applications, the newly added TUNING.md provides guidelines on jemalloc tuning.

New features:

  • Implement transparent huge page support for internal metadata. (@interwq)
  • Add opt.thp to allow enabling / disabling transparent huge pages for all mappings. (@interwq)
  • Add maximum background thread count option. (@djwatson)
  • Allow prof_active to control opt.lg_prof_interval and prof.gdump. (@interwq)
  • Allow arena index lookup based on allocation addresses via mallctl. (@lionkov)
  • Allow disabling initial-exec TLS model. (@davidtgoldblatt, @KenMacD)
  • Add opt.lg_extent_max_active_fit to set the max ratio between the size of the active extent selected (to split off from) and the size of the requested allocation. (@interwq, @davidtgoldblatt)
  • Add retain_grow_limit to set the max size when growing virtual address space. (@interwq)
  • Add mallctl interfaces:

Portability improvements:

Optimizations and refactors:

  • Improve active extent fit with extent_max_active_fit. This considerably reduces fragmentation over time and improves virtual memory and metadata usage. (@davidtgoldblatt, @interwq)
  • Eagerly coalesce large extents to reduce fragmentation. (@interwq)
  • sdallocx: only read size info when page aligned (i.e. possibly sampled), which speeds up the sized deallocation path significantly. (@interwq)
  • Avoid attempting new mappings for in place expansion with retain, since it rarely succeeds in practice and causes high overhead. (@interwq)
  • Refactor OOM handling in newImpl. (@wqfish)
  • Add internal fine-grained logging functionality for debugging use. (@davidtgoldblatt)
  • Refactor arena / tcache interactions. (@davidtgoldblatt)
  • Refactor extent management with dumpable flag. (@davidtgoldblatt)
  • Add runtime detection of lazy purging. (@interwq)
  • Use pairing heap instead of red-black tree for extents_avail. (@djwatson)
  • Use sysctl on startup in FreeBSD. (@trasz)
  • Use thread local prng state instead of atomic. (@djwatson)
  • Make decay to always purge one more extent than before, because in practice large extents are usually the ones that cross the decay threshold. Purging the additional extent helps save memory as well as reduce VM fragmentation. (@interwq)
  • Fast division by dynamic values. (@davidtgoldblatt)
  • Improve the fit for aligned allocation. (@interwq, @edwinsmith)
  • Refactor extent_t bitpacking. (@rkmisra)
  • Optimize the generated assembly for ticker operations. (@davidtgoldblatt)
  • Convert stats printing to use a structured text emitter. (@davidtgoldblatt)
  • Remove preserve_lru feature for extents management. (@djwatson)
  • Consolidate two memory loads into one on the fast deallocation path. (@davidtgoldblatt, @interwq)

Bug fixes (most of the issues are only relevant to jemalloc 5.0):

  • Fix deadlock with multithreaded fork in OS X. (@davidtgoldblatt)
  • Validate returned file descriptor before use. (@zonyitoo)
  • Fix a few background thread initialization and shutdown issues. (@interwq)
  • Fix an extent coalesce + decay race by taking both coalescing extents off the LRU list. (@interwq)
  • Fix potentially unbound increase during decay, caused by one thread keep stashing memory to purge while other threads generating new pages. The number of pages to purge is checked to prevent this. (@interwq)
  • Fix a FreeBSD bootstrap assertion. (@strejda, @interwq)
  • Handle 32 bit mutex counters. (@rkmisra)
  • Fix a indexing bug when creating background threads. (@davidtgoldblatt, @binliu19)
  • Fix arguments passed to extent_init. (@yuleniwo, @interwq)
  • Fix addresses used for ordering mutexes. (@rkmisra)
  • Fix abort_conf processing during bootstrap. (@interwq)
  • Fix include path order for out-of-tree builds. (@cmuellner)

Incompatible changes:

  • Remove --disable-thp. (@interwq)
  • Remove mallctl interfaces:

Documentation:

@jasone jasone released this Jul 2, 2017 · 233 commits to dev since this release

Assets 3

This bugfix release fixes several issues, most of which are obscure enough that typical applications are not impacted.

Bug fixes:

  • Update decay->nunpurged before purging, in order to avoid potential update races and subsequent incorrect purging volume. (@interwq)
  • Only abort on dlsym(3) error if the failure impacts an enabled feature (lazy locking and/or background threads). This mitigates an initialization failure bug for which we still do not have a clear reproduction test case. (@interwq)
  • Modify tsd management so that it neither crashes nor leaks if a thread's only allocation activity is to call free() after TLS destructors have been executed. This behavior was observed when operating with GNU libc, and is unlikely to be an issue with other libc implementations. (@interwq)
  • Mask signals during background thread creation. This prevents signals from being inadvertently delivered to background threads. (@jasone, @davidtgoldblatt, @interwq)
  • Avoid inactivity checks within background threads, in order to prevent recursive mutex acquisition. (@interwq)
  • Fix extent_grow_retained() to use the specified hooks when the arena.<i>.extent_hooks mallctl is used to override the default hooks. (@interwq)
  • Add missing reentrancy support for custom extent hooks which allocate. (@interwq)
  • Post-fork(2), re-initialize the list of tcaches associated with each arena to contain no tcaches except the forking thread's. (@interwq)
  • Add missing post-fork(2) mutex reinitialization for extent_grow_mtx. This fixes potential deadlocks after fork(2). (@interwq)
  • Enforce minimum autoconf version (currently 2.68), since 2.63 is known to generate corrupt configure scripts. (@jasone)
  • Ensure that the configured page size (--with-lg-page) is no larger than the configured huge page size (--with-lg-hugepage). (@jasone)

@jasone jasone released this Jun 13, 2017 · 253 commits to dev since this release

Assets 3

Unlike all previous jemalloc releases, this release does not use naturally aligned "chunks" for virtual memory management, and instead uses page-aligned "extents". This change has few externally visible effects, but the internal impacts are... extensive. Many other internal changes combine to make this the most cohesively designed version of jemalloc so far, with ample opportunity for further enhancements.

Continuous integration is now an integral aspect of development thanks to the efforts of @davidtgoldblatt, and the dev branch tends to remain reasonably stable on the tested platforms (Linux, FreeBSD, macOS, and Windows). As a side effect the official release frequency may decrease over time.

New features:

  • Implement optional per-CPU arena support; threads choose which arena to use based on current CPU rather than on fixed thread-->arena associations. (@interwq)
  • Implement two-phase decay of unused dirty pages. Pages transition from dirty-->muzzy-->clean, where the first phase transition relies on madvise(... MADV_FREE) semantics, and the second phase transition discards pages such that they are replaced with demand-zeroed pages on next access. (@jasone)
  • Increase decay time resolution from seconds to milliseconds. (@jasone)
  • Implement opt-in per CPU background threads, and use them for asynchronous decay-driven unused dirty page purging. (@interwq)
  • Add mutex profiling, which collects a variety of statistics useful for diagnosing overhead/contention issues. (@interwq)
  • Add C++ new/delete operator bindings. (@djwatson)
  • Support manually created arena destruction, such that all data and metadata are discarded. Add MALLCTL_ARENAS_DESTROYED for accessing merged stats associated with destroyed arenas. (@jasone)
  • Add MALLCTL_ARENAS_ALL as a fixed index for use in accessing merged/destroyed arena statistics via mallctl. (@jasone)
  • Add opt.abort_conf to optionally abort if invalid configuration options are detected during initialization. (@interwq)
  • Add opt.stats_print_opts, so that e.g. JSON output can be selected for the stats dumped during exit if opt.stats_print is true. (@jasone)
  • Add --with-version=VERSION for use when embedding jemalloc into another project's git repository. (@jasone)
  • Add --disable-thp to support cross compiling. (@jasone)
  • Add --with-lg-hugepage to support cross compiling. (@jasone)
  • Add mallctl interfaces (various authors):
    • background_thread
    • opt.abort_conf
    • opt.retain
    • opt.percpu_arena
    • opt.background_thread
    • opt.{dirty,muzzy}_decay_ms
    • opt.stats_print_opts
    • arena.<i>.initialized
    • arena.<i>.destroy
    • arena.<i>.{dirty,muzzy}_decay_ms
    • arena.<i>.extent_hooks
    • arenas.{dirty,muzzy}_decay_ms
    • arenas.bin.<i>.slab_size
    • arenas.nlextents
    • arenas.lextent.<i>.size
    • arenas.create
    • stats.background_thread.{num_threads,num_runs,run_interval}
    • stats.mutexes.{ctl,background_thread,prof,reset}.{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,num_owner_switch}
    • stats.arenas.<i>.{dirty,muzzy}_decay_ms
    • stats.arenas.<i>.uptime
    • stats.arenas.<i>.{pmuzzy,base,internal,resident}
    • stats.arenas.<i>.{dirty,muzzy}_{npurge,nmadvise,purged}
    • stats.arenas.<i>.bins.<j>.{nslabs,reslabs,curslabs}
    • stats.arenas.<i>.bins.<j>.mutex.{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,num_owner_switch}
    • stats.arenas.<i>.lextents.<j>.{nmalloc,ndalloc,nrequests,curlextents}
    • stats.arenas.i.mutexes.{large,extent_avail,extents_dirty,extents_muzzy,extents_retained,decay_dirty,decay_muzzy,base,tcache_list}.{num_ops,num_spin_acq,num_wait,max_wait_time,total_wait_time,max_num_thds,num_owner_switch}

Portability improvements:

  • Improve reentrant allocation support, such that deadlock is less likely if e.g. a system library call in turn allocates memory. (@davidtgoldblatt, @interwq)
  • Support static linking of jemalloc with glibc. (@djwatson)

Optimizations and refactors:

  • Organize virtual memory as "extents" of virtual memory pages, rather than as naturally aligned "chunks", and store all metadata in arbitrarily distant locations. This reduces virtual memory external fragmentation, and will interact better with huge pages (not yet explicitly supported). (@jasone)
  • Fold large and huge size classes together; only small and large size classes remain. (@jasone)
  • Unify the allocation paths, and merge most fast-path branching decisions. (@davidtgoldblatt, @interwq)
  • Embed per thread automatic tcache into thread-specific data, which reduces conditional branches and dereferences. Also reorganize tcache to increase fast-path data locality. (@interwq)
  • Rewrite atomics to closely model the C11 API, convert various synchronization from mutex-based to atomic, and use the explicit memory ordering control to resolve various hypothetical races without increasing synchronization overhead. (@davidtgoldblatt)
  • Extensively optimize rtree via various methods:
    • Add multiple layers of rtree lookup caching, since rtree lookups are now part of fast-path deallocation. (@interwq)
    • Determine rtree layout at compile time. (@jasone)
    • Make the tree shallower for common configurations. (@jasone)
    • Embed the root node in the top-level rtree data structure, thus avoiding one level of indirection. (@jasone)
    • Further specialize leaf elements as compared to internal node elements, and directly embed extent metadata needed for fast-path deallocation. (@jasone)
    • Ignore leading always-zero address bits (architecture-specific). (@jasone)
  • Reorganize headers (ongoing work) to make them hermetic, and disentangle various module dependencies. (@davidtgoldblatt)
  • Convert various internal data structures such as size class metadata from boot-time-initialized to compile-time-initialized. Propagate resulting data structure simplifications, such as making arena metadata fixed-size. (@jasone)
  • Simplify size class lookups when constrained to size classes that are multiples of the page size. This speeds lookups, but the primary benefit is complexity reduction in code that was the source of numerous regressions. (@jasone)
  • Lock individual extents when possible for localized extent operations, rather than relying on a top-level arena lock. (@davidtgoldblatt, @jasone)
  • Use first fit layout policy instead of best fit, in order to improve packing. (@jasone)
  • If munmap(2) is not in use, use an exponential series to grow each arena's virtual memory, so that the number of disjoint virtual memory mappings remains low. (@jasone)
  • Implement per arena base allocators, so that arenas never share any virtual memory pages. (@jasone)
  • Automatically generate private symbol name mangling macros. (@jasone)

Incompatible changes:

  • Replace chunk hooks with an expanded/normalized set of extent hooks. (@jasone)
  • Remove ratio-based purging. (@jasone)
  • Remove --disable-tcache. (@jasone)
  • Remove --disable-tls. (@jasone)
  • Remove --enable-ivsalloc. (@jasone)
  • Remove --with-lg-size-class-group. (@jasone)
  • Remove --with-lg-tiny-min. (@jasone)
  • Remove --disable-cc-silence. (@jasone)
  • Remove --enable-code-coverage. (@jasone)
  • Remove --disable-munmap (replaced by opt.retain). (@jasone)
  • Remove Valgrind support. (@jasone)
  • Remove quarantine support. (@jasone)
  • Remove redzone support. (@jasone)
  • Remove mallctl interfaces (various authors):
    • config.munmap
    • config.tcache
    • config.tls
    • config.valgrind
    • opt.lg_chunk
    • opt.purge
    • opt.lg_dirty_mult
    • opt.decay_time
    • opt.quarantine
    • opt.redzone
    • opt.thp
    • arena.<i>.lg_dirty_mult
    • arena.<i>.decay_time
    • arena.<i>.chunk_hooks
    • arenas.initialized
    • arenas.lg_dirty_mult
    • arenas.decay_time
    • arenas.bin.<i>.run_size
    • arenas.nlruns
    • arenas.lrun.<i>.size
    • arenas.nhchunks
    • arenas.hchunk.<i>.size
    • arenas.extend
    • stats.cactive
    • stats.arenas.<i>.lg_dirty_mult
    • stats.arenas.<i>.decay_time
    • stats.arenas.<i>.metadata.{mapped,allocated}
    • stats.arenas.<i>.{npurge,nmadvise,purged}
    • stats.arenas.<i>.huge.{allocated,nmalloc,ndalloc,nrequests}
    • stats.arenas.<i>.bins.<j>.{nruns,reruns,curruns}
    • stats.arenas.<i>.lruns.<j>.{nmalloc,ndalloc,nrequests,curruns}
    • stats.arenas.<i>.hchunks.<j>.{nmalloc,ndalloc,nrequests,curhchunks}

Bug fixes:

  • Improve interval-based profile dump triggering to dump only one profile when a single allocation's size exceeds the interval. (@jasone)
  • Use prefixed function names (as controlled by --with-jemalloc-prefix) when pruning backtrace frames in jeprof. (@jasone)

@jasone jasone released this Mar 1, 2017 · 798 commits to dev since this release

Assets 3

This is the first release to benefit from much broader continuous integration testing, thanks to @davidtgoldblatt. Had we had this testing infrastructure in place for prior releases, it would have caught all of the most serious regressions fixed by this release.

New features:

  • Add --disable-thp and the opt.thp mallctl to provide opt-out mechanisms for transparent huge page integration. (@jasone)
  • Update zone allocator integration to work with macOS 10.12. (@glandium)
  • Restructure *CFLAGS configuration, so that CFLAGS behaves typically, and EXTRA_CFLAGS provides a way to specify e.g. -Werror during building, but not during configuration. (@jasone, @ronawho)

Bug fixes:

  • Fix DSS (sbrk(2)-based) allocation. This regression was first released in 4.3.0. (@jasone)
  • Handle race in per size class utilization computation. This functionality was first released in 4.0.0. (@interwq)
  • Fix lock order reversal during gdump. (@jasone)
  • Fix/refactor tcache synchronization. This regression was first released in 4.0.0. (@jasone)
  • Fix various JSON-formatted malloc_stats_print() bugs. This functionality was first released in 4.3.0. (@jasone)
  • Fix huge-aligned allocation. This regression was first released in 4.4.0. (@jasone)
  • When transparent huge page integration is enabled, detect what state pages start in according to the kernel's current operating mode, and only convert arena chunks to non-huge during purging if that is not their initial state. This functionality was first released in 4.4.0. (@jasone)
  • Fix lg_chunk clamping for the --enable-cache-oblivious --disable-fill case. This regression was first released in 4.0.0. (@jasone, @428desmo)
  • Properly detect sparc64 when building for Linux. (@glaubitz)

@jasone jasone released this Dec 4, 2016 · 826 commits to dev since this release

Assets 3

New features:

  • Add configure support for --linux-android. (@cferris1000, @jasone)
  • Add the --disable-syscall configure option, for use on systems that place security-motivated limitations on syscall(2). (@jasone)
  • Add support for Debian GNU/kFreeBSD. (@thesam)

Optimizations:

  • Add extent serial numbers and use them where appropriate as a sort key that is higher priority than address, so that the allocation policy prefers older extents. This tends to improve locality (decrease fragmentation) when memory grows downward. (@jasone)
  • Refactor madvise(2) configuration so that MADV_FREE is detected and utilized on Linux 4.5 and newer. (@jasone)
  • Mark partially purged arena chunks as non-huge-page. This improves interaction with Linux's transparent huge page functionality. (@jasone)

Bug fixes:

  • Fix size class computations for edge conditions involving extremely large allocations. This regression was first released in 4.0.0. (@jasone, @ingvarha)
  • Remove overly restrictive assertions related to the cactive statistic. This regression was first released in 4.1.0. (@jasone)
  • Implement a more reliable detection scheme for os_unfair_lock on macOS. (@jszakmeister)

@jasone jasone released this Nov 8, 2016 · 858 commits to dev since this release

Assets 3

Bug fixes:

  • Fix a severe virtual memory leak. This regression was first released in 4.3.0. (@interwq, @jasone)
  • Refactor atomic and prng APIs to restore support for 32-bit platforms that use pre-C11 toolchains, e.g. FreeBSD's mips. (@jasone)

@jasone jasone released this Nov 5, 2016 · 863 commits to dev since this release

Assets 3

This is the first release that passes the test suite for multiple Windows configurations, thanks in large part to @glandium setting up continuous integration via AppVeyor (and Travis CI for Linux and OS X).

New features:

  • Add "J" (JSON) support to malloc_stats_print(). (@jasone)
  • Add Cray compiler support. (@ronawho)

Optimizations:

  • Add/use adaptive spinning for bootstrapping and radix tree node initialization. (@jasone)

Bug fixes:

  • Fix large allocation to search starting in the optimal size class heap, which can substantially reduce virtual memory churn and fragmentation. This regression was first released in 4.0.0. (@mjp41, @jasone)
  • Fix stats.arenas.<i>.nthreads accounting. (@interwq)
  • Fix and simplify decay-based purging. (@jasone)
  • Make DSS (sbrk(2)-related) operations lockless, which resolves potential deadlocks during thread exit. (@jasone)
  • Fix over-sized allocation of radix tree leaf nodes. (@mjp41, @ogaun, @jasone)
  • Fix over-sized allocation of arena_t (plus associated stats) data structures. (@jasone, @interwq)
  • Fix EXTRA_CFLAGS to not affect configuration. (@jasone)
  • Fix a Valgrind integration bug. (@ronawho)
  • Disallow 0x5a junk filling when running in Valgrind. (@jasone)
  • Fix a file descriptor leak on Linux. This regression was first released in 4.2.0. (@vsarunas, @jasone)
  • Fix static linking of jemalloc with glibc. (@djwatson)
  • Use syscall(2) rather than {open,read,close}(2) during boot on Linux. This works around other libraries' system call wrappers performing reentrant allocation. (@kspinka, @Whissi, @jasone)
  • Fix OS X default zone replacement to work with OS X 10.12. (@glandium, @jasone)
  • Fix cached memory management to avoid needless commit/decommit operations during purging, which resolves permanent virtual memory map fragmentation issues on Windows. (@mjp41, @jasone)
  • Fix TSD fetches to avoid (recursive) allocation. This is relevant to non-TLS and Windows configurations. (@jasone)
  • Fix malloc_conf overriding to work on Windows. (@jasone)
  • Forcibly disable lazy-lock on Windows (was forcibly enabled). (@jasone)

@jasone jasone released this Jun 8, 2016 · 935 commits to dev since this release

Assets 3

Bug fixes:

  • Fix bootstrapping issues for configurations that require allocation during tsd initialization (e.g. --disable-tls). (@cferris1000, @jasone)
  • Fix gettimeofday() version of nstime_update(). (@ronawho)
  • Fix Valgrind regressions in calloc() and chunk_alloc_wrapper(). (@ronawho)
  • Fix potential VM map fragmentation regression. (@jasone)
  • Fix opt_zero-triggered in-place huge reallocation zeroing. (@jasone)
  • Fix heap profiling context leaks in reallocation edge cases. (@jasone)

@jasone jasone released this May 12, 2016 · 945 commits to dev since this release

Assets 3

New features:

  • Add the arena.<i>.reset mallctl, which makes it possible to discard all of an arena's allocations in a single operation. (@jasone)
  • Add the stats.retained and stats.arenas.<i>.retained statistics. (@jasone)
  • Add the --with-version configure option. (@jasone)
  • Support --with-lg-page values larger than actual page size. (@jasone)

Optimizations:

  • Use pairing heaps rather than red-black trees for various hot data structures. (@djwatson, @jasone)
  • Streamline fast paths of rtree operations. (@jasone)
  • Optimize the fast paths of calloc() and [m,d,sd]allocx(). (@jasone)
  • Decommit unused virtual memory if the OS does not overcommit. (@jasone)
  • Specify MAP_NORESERVE on Linux if [heuristic] overcommit is active, in order to avoid unfortunate interactions during fork(2). (@jasone)

Bug fixes:

  • Fix chunk accounting related to triggering gdump profiles. (@jasone)
  • Link against librt for clock_gettime(2) if glibc < 2.17. (@jasone)
  • Scale leak report summary according to sampling probability. (@jasone)

@jasone jasone released this May 4, 2016 · 1028 commits to dev since this release

Assets 3

This bugfix release resolves a variety of mostly minor issues, though the bitmap fix is critical for 64-bit Windows.

Bug fixes:

  • Fix the linear scan version of bitmap_sfu() to shift by the proper amount even when sizeof(long) is not the same as sizeof(void *), as on 64-bit Windows. (@jasone)
  • Fix hashing functions to avoid unaligned memory accesses (and resulting crashes). This is relevant at least to some ARM-based platforms. (@rkmisra)
  • Fix fork()-related lock rank ordering reversals. These reversals were unlikely to cause deadlocks in practice except when heap profiling was enabled and active. (@jasone)
  • Fix various chunk leaks in OOM code paths. (@jasone)
  • Fix malloc_stats_print() to print opt.narenas correctly. (@jasone)
  • Fix MSVC-specific build/test issues. (@rustyx, @yuslepukhin)
  • Fix a variety of test failures that were due to test fragility rather than core bugs. (@jasone)