New malloc algorithm

https://bugs.webkit.org/show_bug.cgi?id=226404

Reviewed by Yusuke Suzuki.

Source/bmalloc:

This change replaces bmalloc with libpas, by importing libpas and making it expose API that
looks like the current bmalloc API. Libpas replaces everything bmalloc gave us: the main
bmalloc API, the Gigacage API, and the IsoHeap API. Libpas also replaces the JSC
ExecutableAllocator (more on that in the JSC ChangeLog). I've been working on libpas for about
three years, and you'll be able to see the age of some of it in the copyrights.

Libpas is a totally new malloc implementation that focuses on allowing for the creation of lots
of isolated heaps, and making it possible for those heaps to provide certain type guarantees.
I'm not going to summarize everything that it does in this changelog; I'll mostly focus on
things that are interesting to its use in WebKit.

Libpas tries hard to make having lots of heaps cheap both by having a good isolated heap
implementation and by offsetting the cost by having a bunch of space saving improvements for all
kinds of heaps. Libpas heaps can be configured to be either very compact or very fast. In
WebKit, we configure libpas to be as fast as possible, except for the jit_heap, where we
configure it to be as compact as possible. The fast configuration relies on the segregated heap
for most allocations, while the compact configuration relies on bitfit for most configurations.
It's possible to use both in combination with any size cut-off before you switch to bitfit.

Libpas's segregated heap is competitive to bmalloc in speed. This patch is a speed-up on AS, and
it's only enabled on AS for now. Libpas is still a regression relative to bmalloc on some
devices that I have measured, so it should probably stay enabled on AS only for now. This patch
ought to be a 1% speed-up on Speedometer and MotionMark and be neutral elsewhere.

When it comes to memory usage, libpas's segregated heap is a 19% improvement on membuster and a
11% improvement on RAMification on AS, with most other devices exhibiting similar wins if you
enable libpas on them.

Here's the basic design:

- Libpas makes extreme use of generic programming in C, sometimes using macros, but mostly using
  a technique where ALWAYS_INLINE functions are passed structs with function pointers to other
  ALWAYS_INLINE functions, sometimes many layers deep. I've perfected this so that I can cause
  outlining to happen wherever I want without losing the specialization, and I can cause
  anything I want to become generic (i.e. it gets a pointer to that struct with function
  pointers and it really has to call them).

  This allows libpas to have many copies of the same algorithm, but configured differently.
  That's why, for example, I was able to make libpas's bitfit obey ExecutableAllocator rules
  like that you cannot put any metadata inside the memory you're managing. That's also how
  libpas "scales" it's algorithm: there are the small configurations, and then with a different
  struct, we get the medium configurations (larger overall page size, ability to decommit each
  physical page within the logical page), and the marge configurations (optimized for even
  bigger object sizes).

- Allocation uses thread-local caches that hold on to up to a page of memory. This is similar
  overall to bmalloc, but rather different in the details. When allocating from that memory (see
  bmalloc_heap_inlines.h, then pas_try_allocate_intrinsic_primitive.h, then
  pas_try_allocate_common.h, then pas_local_allocator_inlines.h), libpas uses either bump
  pointer or bit search. The bit search is optimized to death. Libpas does lots of clever things
  with bits: for example, each bit represents just a "minalign" worth of memory (so like 16
  bytes), and bits are set at the beginnings of objects. Lots of interesting bit manipulation
  happens on the slow path to quickly turn this into a bitvector where the bits are set at
  free object beginnings, which is what the allocator consumes (see for example
  pas_local_allocator_scan_bits_to_set_up_free_bits() in pas_local_allocator_inlines.h). This
  saves libpas from having to use multiplications on the allocation and deallocation paths.

  Allocation fast paths are super fast and require no locks. Slow paths use fine-grained
  locking so the likelihood of contention is very low. Each page has a lock, and they do crazy
  things with their locks: pages will all share the same lock so long as that's efficient, but
  will pick up different locks if that causes contention, and then they will share the same
  lock again if contention goes away.

  Allocation caches can be reclaimed by the scavenger. This is done using an
  ask-nicely-then-more-firmly kind of mechanism: first the scavenger will just ask allocators
  that haven't been used in a while to return their memory, and the thread will service this
  request on the next allocation slow path for any thread-local allocator. But if the thread
  doesn't do this, then the scavenger will eventually suspend the thread and take its caches.

- Deallocation uses a log. The log flush amortizes lock acquisition and memory access to libpas
  metadata. This is quite a bit like bmalloc. The differences are: libpas may acquire multiple
  locks during deallocation log flush, but because pages try to share locks, that usually
  doesn't happen. Usually, all pages that were most recently used by some CPU will share the
  same lock. The other difference is that the libpas scavenger can flush deallocation logs, and
  does so on every tick.

The design described so far is for the libpas segregated heap, which is the most
performance-optimized heap, as well as the heap most suited for isoheaps. Segregated heap has
many isoheap optimizations, like allowing multiple heaps to share different slabs of the same
page. But libpas also has two other heap kinds:

- Bitfit. The bitfit heap uses a bit-per-minalign but uses it to implement first-fit. Allocation
  searches for a span of set "free" bits and then clears them. Deallocation uses a second
  bitvector, the "object ends" bits, to find the end of the object and then sets the whole range
  of free bits. This allocator uses per-page locking and a clever short-circuiting algorithm
  to find the first page that will have a contiguous slab of free memory big enough for the
  size you're trying to allocate. This is much slower than the segregated heap, but also much
  faster than the other option, which is the libpas large allocator. Bitfit is the most
  space-efficient heap in libpas, except for super large objects. The space efficiency comes
  from the fact that it's first-fit. It should be noted that segregated is also first-fit, but
  within size classes, so segregated gets more external fragmentation than bitfit.

  This patch causes us to use bitfit for the mini mode. We also use bitfit for marge allocations
  (so bigger than segregated medium but too small for large) and for jit_heap.

  I think the most interesting file for bitfit is pas_bitfit_page_inlines.h.

- Large. This is a first-fit allocator implemented using cartesian trees. Something special
  about it is that it can do type-safe array allocations with complex combinations of memalign.
  For example, you can tell it to create a heap that allocates arrays of a type whose size is
  42, and then it will guarantee you that no matter how big or how small of an array you
  allocate, no element of that array will ever be an "offset overlap" with a previously freed
  element. So, it won't create any type confusions. It will guarantee this even if you did
  memalign with any alignment (I needed to use extended GCD for this -- see pas_extended_gcd.c
  and pas_coalign.c). The large algorithm is implemented generically
  (pas_generic_large_free_heap.h), so it can use either cartesian trees
  (pas_fast_large_free_heap.c) or just an array (pas_simple_large_free_heap.c). The large heap
  is used internally in libpas a lot, for example for the bootstrap heap, which is how all of
  libpas gets its memory. One particularly neat thing about this is that the bootstrap heap
  allocates a freelist array inside of the memory it is managing using a kind of wild
  metacircular design.

One of the main space saving features is that libpas has a fine-grained-locked LRU decommit
policy using a 10Hz scavenger thread. This scavenger incurs almost no perf cost but is able
to return a ton of memory. At any tick, it will decommit any totally free pages that haven't
been used in the last 300ms. This includes pages used for internal libpas metadata, like the
thread-local caches. Decommitting pages holds locks that other threads are unlikely to want to
grab (because of policies that actively avoid it), so decommit happens with minimal interference
to the running program.

Libpas is written in C and strongly assumes that the C compiler is modern enough to fixpoint
inlining of always-inline functions with the right other optimizations. Llvm does this, which
enables a wonderful style of generic programming. The fact that the same code can be used
for generic, specialized-and-inline, and specialized-but-out-of-line functions is great. I use
WebKit C++ style (so foo* p rather than foo *p) but with lower_case_identifiers for anything
that isn't a macro. Anything that isn't static is prefixed pas_, unless it's some specific
heap configuration, like the bmalloc one, which prefixes every non-static with bmalloc_.

Libpas includes a giant test suite. Lots of the heap configurations (iso_test, thingy,
minalign32, pagesize64k) are just for testing. The test suite has both unit tests and chaos
tests as well as everything in between. The test suite is written in C++.

This glues libpas into the DebugHeap mechanism in bmalloc, so Malloc=1 will give you system
malloc. Libpas supports libmalloc enumeration, and this patch exposes it (though you need a
shared cache rebuild to enjoy the benefits).

All of bmalloc's external-facing API are replaced with libpas when the BUSE(LIBPAS) flag is
set (so the bmalloc::XYZ functions, Gigacage::XYZ, and IsoHeap<> and friends). This also
exposes the jit_heap API for the ExecutableAllocator to use.

Notes for porting: this uses lots of Darwin APIs, like the APIs for getting an approximate time
very quickly, APIs for fast TLS, thread suspension, and probably other stuff. It'll be hard,
but possible, to port to Linux. It may never perform as well on Linux as it does on Darwin.
It's also worth noting that libpas strongly assumes 64-bit, that the CPU can do 128-bit
compare-and-swap, that there is no big downside to misaligned loads and stores, and that the
memory model is either the one in ARM64 or the one on x86_64 (or something not weaker than
either of them). Trying to make libpas work on 32-bit CPUs will quickly run into cases where
some pointer math doesn't work right because it was written strongly assuming properties
unique to 64-bit CPUs (like that some number of top bits are not used for the address), as well
as cases where lock-free algorithms are using uintptr_t for a versioning scheme (which may not
be as reliable as you want on 32-bit).

* CMakeLists.txt:
* Configurations/Base.xcconfig:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/BPlatform.h:
* bmalloc/DebugHeap.cpp:
(bmalloc::DebugHeap::tryGetSlow):
(pas_debug_heap_is_enabled):
(pas_debug_heap_malloc):
(pas_debug_heap_memalign):
(pas_debug_heap_realloc):
(pas_debug_heap_free):
* bmalloc/DebugHeap.h:
(bmalloc::debugHeapDisabled):
(bmalloc::DebugHeap::tryGet):
(bmalloc::DebugHeap::getExisting):
* bmalloc/Gigacage.cpp:
(Gigacage::ensureGigacage):
(Gigacage::allocBase):
(Gigacage::size):
* bmalloc/Gigacage.h:
* bmalloc/GigacageConfig.h:
(Gigacage::Config::allocBasePtr const):
(Gigacage::Config::setAllocBasePtr):
(Gigacage::Config::allocSize const):
(Gigacage::Config::setAllocSize):
* bmalloc/Heap.cpp:
(bmalloc::Heap::gigacageSize):
* bmalloc/Heap.h:
* bmalloc/IsoHeap.cpp: Added.
(bmalloc::api::isoAllocate):
(bmalloc::api::isoTryAllocate):
(bmalloc::api::isoDeallocate):
* bmalloc/IsoHeap.h:
(bmalloc::api::IsoHeap::IsoHeap):
(bmalloc::api::IsoHeap::allocate):
(bmalloc::api::IsoHeap::tryAllocate):
(bmalloc::api::IsoHeap::deallocate):
(bmalloc::api::IsoHeap::scavenge):
(bmalloc::api::IsoHeap::initialize):
(bmalloc::api::IsoHeap::isInitialized):
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
* bmalloc/IsoHeapInlines.h:
* bmalloc/IsoMallocFallback.cpp: Added.
(bmalloc::IsoMallocFallback::tryMalloc):
(bmalloc::IsoMallocFallback::tryFree):
* bmalloc/IsoMallocFallback.h: Copied from Source/bmalloc/bmalloc/DebugHeap.h.
(bmalloc::IsoMallocFallback::shouldTryToFallBack):
(bmalloc::IsoMallocFallback::MallocResult::MallocResult):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::determineMallocFallbackState): Deleted.
* bmalloc/IsoTLS.h:
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/PerThread.h:
(bmalloc::PerThreadStorage<PerHeapKind<Cache>>::init):
* bmalloc/bmalloc.cpp:
(bmalloc::api::tryLargeZeroedMemalignVirtual):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::commitAlignedPhysical):
(bmalloc::api::decommitAlignedPhysical):
(bmalloc::api::enableMiniMode):
(bmalloc::api::disableScavenger):
* bmalloc/bmalloc.h:
(bmalloc::api::heapForKind):
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::tryRealloc):
(bmalloc::api::realloc):
(bmalloc::api::free):
(bmalloc::api::scavengeThisThread): Deleted.
* libpas/.gitignore: Added.
* libpas/build.sh: Added.
* libpas/build_and_test.sh: Added.
* libpas/common.sh: Added.
* libpas/libpas.xcodeproj/project.pbxproj: Added.
* libpas/scripts/tally_verifier_output: Added.
* libpas/src/chaos/Chaos.cpp: Added.
(std::Object::Object):
(std::Packet::~Packet):
(std::Locker::Locker):
(std::Locker::~Locker):
(std::threadMain):
(main):
* libpas/src/libpas/bmalloc_heap.c: Added.
(bmalloc_try_allocate):
(bmalloc_try_allocate_with_alignment):
(bmalloc_try_allocate_zeroed):
(bmalloc_allocate):
(bmalloc_allocate_with_alignment):
(bmalloc_allocate_zeroed):
(bmalloc_try_reallocate):
(bmalloc_reallocate):
(bmalloc_try_iso_allocate):
(bmalloc_iso_allocate):
(bmalloc_heap_ref_get_heap):
(bmalloc_try_allocate_auxiliary):
(bmalloc_allocate_auxiliary):
(bmalloc_try_allocate_auxiliary_zeroed):
(bmalloc_allocate_auxiliary_zeroed):
(bmalloc_try_allocate_auxiliary_with_alignment):
(bmalloc_allocate_auxiliary_with_alignment):
(bmalloc_try_reallocate_auxiliary):
(bmalloc_reallocate_auxiliary):
(bmalloc_deallocate):
(bmalloc_force_auxiliary_heap_into_reserved_memory):
* libpas/src/libpas/bmalloc_heap.h: Added.
* libpas/src/libpas/bmalloc_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(bmalloc_heap_config_activate):
* libpas/src/libpas/bmalloc_heap_config.h: Added.
* libpas/src/libpas/bmalloc_heap_inlines.h: Added.
(bmalloc_try_allocate_inline):
(bmalloc_try_allocate_with_alignment_inline):
(bmalloc_try_allocate_zeroed_inline):
(bmalloc_allocate_inline):
(bmalloc_allocate_with_alignment_inline):
(bmalloc_allocate_zeroed_inline):
(bmalloc_try_reallocate_inline):
(bmalloc_reallocate_inline):
(bmalloc_try_iso_allocate_inline):
(bmalloc_iso_allocate_inline):
(bmalloc_try_allocate_auxiliary_inline):
(bmalloc_allocate_auxiliary_inline):
(bmalloc_try_allocate_auxiliary_zeroed_inline):
(bmalloc_allocate_auxiliary_zeroed_inline):
(bmalloc_try_allocate_auxiliary_with_alignment_inline):
(bmalloc_allocate_auxiliary_with_alignment_inline):
(bmalloc_try_reallocate_auxiliary_inline):
(bmalloc_reallocate_auxiliary_inline):
(bmalloc_deallocate_inline):
* libpas/src/libpas/bmalloc_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/bmalloc_heap_ref.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/hotbit_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(hotbit_try_allocate):
(hotbit_try_allocate_with_alignment):
(hotbit_try_reallocate):
(hotbit_deallocate):
* libpas/src/libpas/hotbit_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/hotbit_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(hotbit_heap_config_activate):
* libpas/src/libpas/hotbit_heap_config.h: Added.
* libpas/src/libpas/hotbit_heap_inlines.h: Added.
(hotbit_try_allocate_inline):
(hotbit_try_allocate_with_alignment_inline):
(hotbit_try_reallocate_inline):
(hotbit_deallocate_inline):
* libpas/src/libpas/hotbit_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap.c: Added.
(iso_try_allocate_common_primitive):
(iso_try_allocate_common_primitive_with_alignment):
(iso_try_allocate_common_primitive_zeroed):
(iso_allocate_common_primitive):
(iso_allocate_common_primitive_with_alignment):
(iso_allocate_common_primitive_zeroed):
(iso_try_reallocate_common_primitive):
(iso_reallocate_common_primitive):
(iso_try_allocate_dynamic_primitive):
(iso_try_allocate_dynamic_primitive_with_alignment):
(iso_try_allocate_dynamic_primitive_zeroed):
(iso_try_reallocate_dynamic_primitive):
(iso_heap_ref_construct):
(iso_try_allocate):
(iso_allocate):
(iso_try_allocate_array):
(iso_allocate_array):
(iso_try_allocate_array_zeroed):
(iso_allocate_array_zeroed):
(iso_try_reallocate_array):
(iso_reallocate_array):
(iso_heap_ref_get_heap):
(iso_primitive_heap_ref_construct):
(iso_try_allocate_primitive):
(iso_allocate_primitive):
(iso_try_allocate_primitive_zeroed):
(iso_allocate_primitive_zeroed):
(iso_try_allocate_primitive_with_alignment):
(iso_allocate_primitive_with_alignment):
(iso_try_reallocate_primitive):
(iso_reallocate_primitive):
(iso_try_allocate_for_objc):
(iso_has_object):
(iso_get_allocation_size):
(iso_get_heap):
(iso_deallocate):
(iso_force_primitive_heap_into_reserved_memory):
* libpas/src/libpas/iso_heap.h: Added.
* libpas/src/libpas/iso_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap_config.h: Added.
* libpas/src/libpas/iso_heap_inlines.h: Added.
(iso_try_allocate_common_primitive_inline):
(iso_try_allocate_common_primitive_with_alignment_inline):
(iso_try_allocate_common_primitive_zeroed_inline):
(iso_allocate_common_primitive_inline):
(iso_allocate_common_primitive_with_alignment_inline):
(iso_allocate_common_primitive_zeroed_inline):
(iso_try_reallocate_common_primitive_inline):
(iso_reallocate_common_primitive_inline):
(iso_try_allocate_inline):
(iso_allocate_inline):
(iso_try_allocate_array_inline):
(iso_allocate_array_inline):
(iso_try_allocate_array_zeroed_inline):
(iso_allocate_array_zeroed_inline):
(iso_try_reallocate_array_inline):
(iso_reallocate_array_inline):
(iso_try_allocate_primitive_inline):
(iso_allocate_primitive_inline):
(iso_try_allocate_primitive_zeroed_inline):
(iso_allocate_primitive_zeroed_inline):
(iso_try_allocate_primitive_with_alignment_inline):
(iso_allocate_primitive_with_alignment_inline):
(iso_try_reallocate_primitive_inline):
(iso_reallocate_primitive_inline):
(iso_try_allocate_for_objc_inline):
(iso_has_object_inline):
(iso_get_allocation_size_inline):
(iso_get_heap_inline):
(iso_deallocate_inline):
* libpas/src/libpas/iso_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap_ref.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap.c: Added.
(iso_test_allocate_common_primitive):
(iso_test_allocate):
(iso_test_allocate_array):
(iso_test_deallocate):
(iso_test_heap_ref_get_heap):
* libpas/src/libpas/iso_test_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap_config.h: Added.
* libpas/src/libpas/jit_heap.c: Added.
(jit_heap_add_fresh_memory):
(jit_heap_try_allocate):
(jit_heap_shrink):
(jit_heap_get_size):
(jit_heap_deallocate):
* libpas/src/libpas/jit_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/jit_heap_config.c: Added.
(fresh_memory_aligned_allocator):
(initialize_fresh_memory_config):
(allocate_from_fresh):
(page_provider):
(jit_page_header_for_boundary_remote):
(jit_small_bitfit_allocate_page):
(jit_small_bitfit_create_page_header):
(jit_small_bitfit_destroy_page_header):
(jit_medium_bitfit_allocate_page):
(jit_medium_bitfit_create_page_header):
(jit_medium_bitfit_destroy_page_header):
(jit_aligned_allocator):
(jit_prepare_to_enumerate):
(jit_heap_config_for_each_shared_page_directory):
(jit_heap_config_for_each_shared_page_directory_remote):
(jit_heap_config_add_fresh_memory):
* libpas/src/libpas/jit_heap_config.h: Added.
(jit_type_size):
(jit_type_alignment):
(jit_heap_config_fast_megapage_kind):
(jit_small_bitfit_page_header_for_boundary):
(jit_small_bitfit_boundary_for_page_header):
(jit_medium_bitfit_page_header_for_boundary):
(jit_medium_bitfit_boundary_for_page_header):
(jit_heap_config_page_header):
* libpas/src/libpas/jit_heap_config_root_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/minalign32_heap.c: Added.
(minalign32_allocate_common_primitive):
(minalign32_allocate):
(minalign32_allocate_array):
(minalign32_deallocate):
(minalign32_heap_ref_get_heap):
* libpas/src/libpas/minalign32_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/minalign32_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(minalign32_heap_config_activate):
* libpas/src/libpas/minalign32_heap_config.h: Added.
* libpas/src/libpas/pagesize64k_heap.c: Added.
(pagesize64k_allocate_common_primitive):
(pagesize64k_allocate):
(pagesize64k_allocate_array):
(pagesize64k_deallocate):
(pagesize64k_heap_ref_get_heap):
* libpas/src/libpas/pagesize64k_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pagesize64k_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(pagesize64k_heap_config_activate):
* libpas/src/libpas/pagesize64k_heap_config.h: Added.
* libpas/src/libpas/pas_aligned_allocation_result.h: Added.
(pas_aligned_allocation_result_create_empty):
(pas_aligned_allocation_result_as_allocation_result):
* libpas/src/libpas/pas_aligned_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_alignment.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_alignment_dump):
* libpas/src/libpas/pas_alignment.h: Added.
(pas_alignment_create):
(pas_alignment_create_traditional):
(pas_alignment_create_trivial):
(pas_alignment_validate):
(pas_alignment_is_ptr_aligned):
(pas_alignment_round_up):
(pas_alignment_is_equal):
* libpas/src/libpas/pas_all_biasing_directories.c: Added.
(pas_all_biasing_directories_append):
(pas_all_biasing_directories_activate):
(scavenge_for_each_set_bit_bits_source):
(scavenge_for_each_set_bit_callback):
(scavenge_bitvector_word_callback):
(pas_all_biasing_directories_scavenge):
* libpas/src/libpas/pas_all_biasing_directories.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_heap_configs.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_heaps.c: Added.
(pas_all_heaps_add_heap):
(pas_all_heaps_for_each_static_heap):
(pas_all_heaps_for_each_dynamic_heap):
(pas_all_heaps_for_each_heap):
(pas_all_heaps_for_each_static_segregated_heap_not_part_of_a_heap):
(for_each_segregated_heap_callback):
(pas_all_heaps_for_each_static_segregated_heap):
(pas_all_heaps_for_each_segregated_heap):
(get_num_free_bytes_for_each_heap_callback):
(pas_all_heaps_get_num_free_bytes):
(reset_heap_ref_for_each_heap_callback):
(pas_all_heaps_reset_heap_ref):
(for_each_segregated_directory_global_size_directory_callback):
(for_each_segregated_directory_shared_page_directory_callback):
(for_each_segregated_directory_segregated_heap_callback):
(pas_all_heaps_for_each_segregated_directory):
(dump_directory_nicely):
(dump_view_nicely):
(verify_in_steady_state_segregated_directory_callback):
(pas_all_heaps_verify_in_steady_state):
(compute_total_non_utility_segregated_summary_directory_callback):
(pas_all_heaps_compute_total_non_utility_segregated_summary):
(compute_total_non_utility_bitfit_summary_heap_callback):
(pas_all_heaps_compute_total_non_utility_bitfit_summary):
(compute_total_non_utility_large_summary_heap_callback):
(pas_all_heaps_compute_total_non_utility_large_summary):
(pas_all_heaps_compute_total_non_utility_summary):
* libpas/src/libpas/pas_all_heaps.h: Added.
* libpas/src/libpas/pas_all_magazines.c: Added.
(pas_all_magazines_get_current):
* libpas/src/libpas/pas_all_magazines.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_shared_page_directories.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_all_shared_page_directories_add):
(pas_all_shared_page_directories_for_each):
* libpas/src/libpas/pas_all_shared_page_directories.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_callbacks.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_callbacks.h: Added.
(pas_did_allocate):
(pas_will_deallocate):
* libpas/src/libpas/pas_allocation_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocation_kind_get_string):
* libpas/src/libpas/pas_allocation_result.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocation_result_zero):
* libpas/src/libpas/pas_allocation_result.h: Added.
(pas_allocation_result_create_failure):
(pas_allocation_result_create_success_with_zero_mode):
(pas_allocation_result_create_success):
* libpas/src/libpas/pas_allocator_counts.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocator_index.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocator_scavenge_action.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocator_scavenge_action_get_string):
* libpas/src/libpas/pas_baseline_allocator.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_baseline_allocator_attach_directory):
(pas_baseline_allocator_detach_directory):
* libpas/src/libpas/pas_baseline_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_baseline_allocator_result.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_baseline_allocator_result_create_failure):
(pas_baseline_allocator_result_create_success):
* libpas/src/libpas/pas_baseline_allocator_table.c: Added.
(initialize):
(pas_baseline_allocator_table_initialize_if_necessary):
(pas_baseline_allocator_table_get_random_index):
(pas_baseline_allocator_table_for_all):
* libpas/src/libpas/pas_baseline_allocator_table.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_config_enumerator_data.c: Added.
(pas_basic_heap_config_enumerator_data_add_page_header_table):
* libpas/src/libpas/pas_basic_heap_config_enumerator_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_config_root_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_page_caches.h: Added.
(pas_basic_heap_page_caches_get_shared_page_directories):
* libpas/src/libpas/pas_basic_heap_runtime_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_segregated_page_caches.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_biasing_directory.c: Added.
(pas_biasing_directory_construct):
(pas_biasing_directory_get_sharing_pool):
(pas_biasing_directory_did_create_delta):
(pas_biasing_directory_take_last_unused):
(pas_biasing_directory_did_use_index_slow):
(pas_biasing_directory_index_did_become_eligible):
* libpas/src/libpas/pas_biasing_directory.h: Added.
(pas_is_segregated_biasing_directory):
(pas_is_bitfit_biasing_directory):
(pas_biasing_directory_magazine_index):
(pas_biasing_directory_ownership_threshold):
(pas_biasing_directory_unused_span_size):
(pas_biasing_directory_did_use_index):
* libpas/src/libpas/pas_biasing_directory_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_unwrap_segregated_biasing_directory):
(pas_unwrap_bitfit_biasing_directory):
* libpas/src/libpas/pas_biasing_directory_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_biasing_directory_kind_get_string):
* libpas/src/libpas/pas_bitfield_vector.h: Added.
(pas_bitfield_vector_get):
(pas_bitfield_vector_set):
* libpas/src/libpas/pas_bitfit_allocation_result.h: Added.
(pas_bitfit_allocation_result_create_success):
(pas_bitfit_allocation_result_create_failure):
(pas_bitfit_allocation_result_create_empty):
(pas_bitfit_allocation_result_create_need_to_lock_commit_lock):
* libpas/src/libpas/pas_bitfit_allocator.c: Added.
(pas_bitfit_allocator_commit_view):
(pas_bitfit_allocator_finish_failing):
* libpas/src/libpas/pas_bitfit_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_allocator_inlines.h: Added.
(pas_bitfit_allocator_reset):
(pas_bitfit_allocator_assert_reset):
(pas_bitfit_allocator_try_allocate):
* libpas/src/libpas/pas_bitfit_biasing_directory.c: Added.
(pas_bitfit_biasing_directory_create):
(pas_bitfit_biasing_directory_take_last_unused):
* libpas/src/libpas/pas_bitfit_biasing_directory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_directory.c: Added.
(pas_bitfit_directory_construct):
(pas_bitfit_directory_update_biasing_eligibility):
(pas_bitfit_directory_max_free_did_become_unprocessed):
(pas_bitfit_directory_max_free_did_become_unprocessed_unchecked):
(pas_bitfit_directory_max_free_did_become_empty_without_biasing_update):
(pas_bitfit_directory_max_free_did_become_empty):
(pas_bitfit_directory_get_first_free_view):
(pas_bitfit_directory_compute_summary):
(for_each_live_object_callback):
(pas_bitfit_directory_for_each_live_object):
* libpas/src/libpas/pas_bitfit_directory.h: Added.
(pas_bitfit_directory_size):
(pas_bitfit_directory_get_max_free_ptr):
(pas_bitfit_directory_get_max_free):
(pas_bitfit_directory_set_max_free_unchecked):
(pas_bitfit_directory_set_max_free_not_empty_impl):
(pas_bitfit_directory_set_processed_max_free):
(pas_bitfit_directory_set_unprocessed_max_free_unchecked):
(pas_bitfit_directory_set_empty_max_free):
(pas_bitfit_directory_set_unprocessed_max_free):
(pas_bitfit_directory_get_view_ptr):
(pas_bitfit_directory_get_view):
* libpas/src/libpas/pas_bitfit_directory_and_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_directory_and_index_create):
* libpas/src/libpas/pas_bitfit_directory_inlines.h: Added.
(pas_bitfit_directory_get_global):
(pas_bitfit_directory_find_first_free_for_num_bits_iterate_callback):
(pas_bitfit_directory_find_first_free_for_num_bits):
(pas_bitfit_directory_find_first_free):
(pas_bitfit_directory_find_first_empty_iterate_callback):
(pas_bitfit_directory_find_first_empty):
* libpas/src/libpas/pas_bitfit_directory_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_directory_kind_get_string):
* libpas/src/libpas/pas_bitfit_global_directory.c: Added.
(pas_bitfit_global_directory_construct):
(pas_bitfit_global_directory_does_sharing):
(pas_bitfit_global_directory_get_use_epoch):
(pas_bitfit_global_directory_get_empty_bit_at_index):
(pas_bitfit_global_directory_set_empty_bit_at_index):
(pas_bitfit_global_directory_view_did_become_empty_at_index):
(pas_bitfit_global_directory_view_did_become_empty):
(pas_bitfit_global_directory_take_last_empty):
(pas_bitfit_global_directory_dump_reference):
(pas_bitfit_global_directory_dump_for_spectrum):
* libpas/src/libpas/pas_bitfit_global_directory.h: Added.
* libpas/src/libpas/pas_bitfit_global_size_class.c: Added.
(pas_bitfit_global_size_class_create):
(pas_bitfit_global_size_class_select_for_magazine):
* libpas/src/libpas/pas_bitfit_global_size_class.h: Added.
* libpas/src/libpas/pas_bitfit_heap.c: Added.
(pas_bitfit_heap_create):
(pas_bitfit_heap_select_variant):
(pas_bitfit_heap_ensure_global_size_class):
(pas_bitfit_heap_compute_summary):
(for_each_live_object_callback):
(pas_bitfit_heap_for_each_live_object):
* libpas/src/libpas/pas_bitfit_heap.h: Added.
(pas_bitfit_heap_get_directory):
* libpas/src/libpas/pas_bitfit_max_free.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page.c: Added.
(pas_bitfit_page_construct):
(pas_bitfit_page_get_config):
(pas_bitfit_page_log_bits):
(pas_bitfit_page_deallocation_did_fail):
(pas_bitfit_page_for_each_live_object):
(verify_for_each_object_callback):
(pas_bitfit_page_verify):
* libpas/src/libpas/pas_bitfit_page.h: Added.
(pas_bitfit_page_header_size):
(pas_bitfit_page_free_bits):
(pas_bitfit_page_object_end_bits):
(pas_bitfit_page_get_granule_use_counts):
(pas_bitfit_page_offset_to_first_object):
(pas_bitfit_page_offset_to_end_of_last_object):
(pas_bitfit_page_payload_size):
(pas_bitfit_page_for_boundary):
(pas_bitfit_page_for_boundary_or_null):
(pas_bitfit_page_for_boundary_unchecked):
(pas_bitfit_page_boundary):
(pas_bitfit_page_boundary_or_null):
(pas_bitfit_page_for_address_and_page_config):
(pas_bitfit_page_testing_verify):
* libpas/src/libpas/pas_bitfit_page_config.h: Added.
(pas_bitfit_page_config_is_enabled):
(pas_bitfit_page_config_num_alloc_bits):
(pas_bitfit_page_config_num_alloc_words):
(pas_bitfit_page_config_num_alloc_words64):
(pas_bitfit_page_config_num_alloc_bit_bytes):
(pas_bitfit_page_config_byte_offset_for_object_bits):
(pas_bitfit_page_config_uses_subpages):
* libpas/src/libpas/pas_bitfit_page_config_inlines.h: Added.
* libpas/src/libpas/pas_bitfit_page_config_kind.c: Added.
(pas_bitfit_page_config_kind_for_each):
* libpas/src/libpas/pas_bitfit_page_config_kind.def: Added.
* libpas/src/libpas/pas_bitfit_page_config_kind.h: Added.
(pas_bitfit_page_config_kind_get_string):
(pas_bitfit_page_config_kind_get_config):
* libpas/src/libpas/pas_bitfit_page_config_utils.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page_config_utils_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page_config_variant.h: Added.
(pas_bitfit_page_config_variant_get_string):
(pas_bitfit_page_config_variant_get_capitalized_string):
* libpas/src/libpas/pas_bitfit_page_inlines.h: Added.
(pas_bitfit_page_compute_offset):
(pas_bitfit_page_allocation_satisfies_alignment):
(pas_bitfit_page_allocation_commit_granules_or_reloop):
(pas_bitfit_page_finish_allocation):
(pas_bitfit_page_allocate):
(pas_bitfit_page_deallocate_with_page_impl_mode_get_string):
(pas_bitfit_page_deallocate_with_page_impl):
(pas_bitfit_page_deallocate_with_page):
(pas_bitfit_page_deallocate):
(pas_bitfit_page_get_allocation_size_with_page):
(pas_bitfit_page_get_allocation_size):
(pas_bitfit_page_shrink_with_page):
(pas_bitfit_page_shrink):
* libpas/src/libpas/pas_bitfit_size_class.c: Added.
(pas_bitfit_size_class_find_insertion_point):
(pas_bitfit_size_class_construct):
(pas_bitfit_size_class_create):
(pas_bitfit_size_class_get_first_free_view):
* libpas/src/libpas/pas_bitfit_size_class.h: Added.
* libpas/src/libpas/pas_bitfit_view.c: Added.
(pas_bitfit_view_create):
(pas_bitfit_view_lock_ownership_lock_slow):
(pas_bitfit_view_note_nonemptiness):
(did_become_empty_for_bits):
(pas_bitfit_view_note_full_emptiness):
(pas_bitfit_view_note_partial_emptiness):
(pas_bitfit_view_note_max_free):
(compute_summary):
(pas_bitfit_view_compute_summary):
(for_each_live_object_callback):
(for_each_live_object):
(pas_bitfit_view_for_each_live_object):
* libpas/src/libpas/pas_bitfit_view.h: Added.
(pas_bitfit_view_lock_ownership_lock):
* libpas/src/libpas/pas_bitfit_view_and_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_view_and_index_create):
(pas_bitfit_view_and_index_create_empty):
* libpas/src/libpas/pas_bitfit_view_inlines.h: Added.
(pas_bitfit_view_current_directory_and_index):
(pas_bitfit_view_current_directory):
(pas_bitfit_view_index_in_current):
(pas_bitfit_view_is_empty):
* libpas/src/libpas/pas_bitvector.h: Added.
(pas_bitvector_get_from_word):
(pas_bitvector_get):
(pas_bitvector_get_from_one_word):
(pas_backward_bitvector_get):
(pas_bitvector_set_in_word):
(pas_bitvector_set):
(pas_bitvector_set_in_one_word):
(pas_bitvector_set_atomic_in_word):
(pas_bitvector_set_atomic):
(pas_backward_bitvector_set):
(pas_bitvector_find_first_set):
(pas_bitvector_for_each_set_bit):
(pas_bitvector64_set_range):
* libpas/src/libpas/pas_bootstrap_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(bootstrap_source_allocate_aligned):
(initialize_config):
* libpas/src/libpas/pas_bootstrap_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bootstrap_heap_page_provider.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bootstrap_heap_page_provider):
* libpas/src/libpas/pas_bootstrap_heap_page_provider.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_cares_about_size_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_cartesian_tree.h: Added.
(pas_cartesian_tree_node_child_ptr):
(pas_cartesian_tree_node_is_null_constrained):
(pas_cartesian_tree_node_minimum):
(pas_cartesian_tree_node_minimum_constrained):
(pas_cartesian_tree_node_maximum):
(pas_cartesian_tree_node_maximum_constrained):
(pas_cartesian_tree_node_successor):
(pas_cartesian_tree_node_successor_constrained):
(pas_cartesian_tree_node_predecessor):
(pas_cartesian_tree_node_predecessor_constrained):
(pas_cartesian_tree_node_reset):
(pas_cartesian_tree_construct):
(pas_cartesian_tree_node_find_exact):
(pas_cartesian_tree_find_exact):
(pas_cartesian_tree_node_find_least_greater_than_or_equal):
(pas_cartesian_tree_find_least_greater_than_or_equal):
(pas_cartesian_tree_node_find_least_greater_than):
(pas_cartesian_tree_find_least_greater_than):
(pas_cartesian_tree_node_find_greatest_less_than_or_equal):
(pas_cartesian_tree_find_greatest_less_than_or_equal):
(pas_cartesian_tree_node_find_greatest_less_than):
(pas_cartesian_tree_find_greatest_less_than):
(pas_cartesian_tree_minimum):
(pas_cartesian_tree_minimum_constrained):
(pas_cartesian_tree_maximum):
(pas_cartesian_tree_maximum_constrained):
(pas_cartesian_tree_is_empty):
(pas_cartesian_tree_insert):
(pas_cartesian_tree_remove):
(pas_cartesian_tree_validate_recurse):
(pas_cartesian_tree_validate):
(pas_cartesian_tree_size):
* libpas/src/libpas/pas_coalign.c: Added.
(formal_mod):
(ceiling_div):
(pas_coalign_one_sided):
(pas_coalign):
* libpas/src/libpas/pas_coalign.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_coalign_empty_result):
(pas_coalign_result_create):
* libpas/src/libpas/pas_commit_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_commit_mode_get_string):
* libpas/src/libpas/pas_commit_span.c: Added.
(pas_commit_span_construct):
(pas_commit_span_add_to_change):
(pas_commit_span_add_unchanged):
(commit):
(pas_commit_span_add_unchanged_and_commit):
(decommit):
(pas_commit_span_add_unchanged_and_decommit):
* libpas/src/libpas/pas_commit_span.h: Added.
* libpas/src/libpas/pas_compact_atomic_allocator_index_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_global_size_class_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_size_class_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_enumerable_range_list_chunk_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_page_sharing_pool_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_ptr.h: Added.
* libpas/src/libpas/pas_compact_atomic_segregated_exclusive_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_global_size_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_heap_page_sharing_pools_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_partial_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_view.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_thread_local_cache_layout_node.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_global_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bootstrap_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(compact_bootstrap_source_allocate_aligned):
(initialize_config):
* libpas/src/libpas/pas_compact_bootstrap_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_cartesian_tree_node_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_heap_reservation.c: Added.
(pas_compact_heap_reservation_try_allocate):
* libpas/src/libpas/pas_compact_heap_reservation.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_large_utility_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_large_utility_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_page_granule_use_count_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_ptr.h: Added.
* libpas/src/libpas/pas_compact_segregated_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_global_size_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_shared_page_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_shared_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_view.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_skip_list_node_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_skip_list_node_ptr_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_subpage_map_entry_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_tagged_atomic_ptr.h: Added.
* libpas/src/libpas/pas_compact_tagged_page_granule_use_count_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_tagged_ptr.h: Added.
* libpas/src/libpas/pas_compact_tagged_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compute_summary_object_callbacks.c: Added.
(pas_compute_summary_live_object_callback):
(pas_compute_summary_live_object_callback_without_physical_sharing):
(config):
(pas_compute_summary_dead_object_callback):
(pas_compute_summary_dead_object_callback_without_physical_sharing):
* libpas/src/libpas/pas_compute_summary_object_callbacks.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_config.h: Added.
* libpas/src/libpas/pas_config_prefix.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_count_lookup_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_count_lookup_mode_get_string):
* libpas/src/libpas/pas_create_basic_heap_page_caches_with_reserved_memory.c: Added.
(pas_create_basic_heap_page_caches_with_reserved_memory):
* libpas/src/libpas/pas_create_basic_heap_page_caches_with_reserved_memory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deallocate.c: Added.
(pas_try_deallocate_known_large):
(pas_deallocate_known_large):
(pas_try_deallocate_slow):
(deallocate_segregated):
(pas_try_deallocate_slow_no_cache):
* libpas/src/libpas/pas_deallocate.h: Added.
(pas_deallocate_known_segregated_impl):
(pas_deallocate_known_segregated):
(pas_try_deallocate_not_small):
(pas_try_deallocate_impl):
(pas_try_deallocate):
(pas_deallocate):
* libpas/src/libpas/pas_deallocation_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_deallocation_mode_get_string):
* libpas/src/libpas/pas_deallocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deallocator_scavenge_action.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_deallocator_scavenge_action_get_string):
* libpas/src/libpas/pas_debug_heap.h: Added.
(pas_debug_heap_is_enabled):
(pas_debug_heap_malloc):
(pas_debug_heap_memalign):
(pas_debug_heap_realloc):
(pas_debug_heap_free):
* libpas/src/libpas/pas_debug_spectrum.c: Added.
(pas_debug_spectrum_add):
(pas_debug_spectrum_dump):
(pas_debug_spectrum_reset):
* libpas/src/libpas/pas_debug_spectrum.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deferred_decommit_log.c: Added.
(pas_deferred_decommit_log_construct):
(pas_deferred_decommit_log_destruct):
(already_holds_lock):
(pas_deferred_decommit_log_lock_for_adding):
(pas_deferred_decommit_log_add):
(pas_deferred_decommit_log_add_already_locked):
(pas_deferred_decommit_log_add_maybe_locked):
(pas_deferred_decommit_log_unlock_after_aborted_add):
(decommit_all):
(pas_deferred_decommit_log_decommit_all):
(pas_deferred_decommit_log_pretend_to_decommit_all):
* libpas/src/libpas/pas_deferred_decommit_log.h: Added.
* libpas/src/libpas/pas_designated_intrinsic_heap.c: Added.
(set_up_range):
(pas_designated_intrinsic_heap_initialize):
* libpas/src/libpas/pas_designated_intrinsic_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_designated_intrinsic_heap_inlines.h: Added.
(pas_designated_intrinsic_heap_num_allocator_indices):
(pas_designated_index_result_create_failure):
(pas_designated_index_result_create_success):
(pas_designated_intrinsic_heap_num_designated_indices):
(pas_designated_intrinsic_heap_designated_index):
* libpas/src/libpas/pas_dyld_state.c: Added.
(pas_dyld_is_libsystem_initialized):
* libpas/src/libpas/pas_dyld_state.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_dynamic_primitive_heap_map.c: Added.
(pas_dynamic_primitive_heap_map_find_slow):
* libpas/src/libpas/pas_dynamic_primitive_heap_map.h: Added.
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_create_empty):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_create_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_empty_or_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_empty):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_get_key):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_key_get_hash):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_key_is_equal):
(pas_dynamic_primitive_heap_map_hash):
(pas_dynamic_primitive_heap_map_find):
* libpas/src/libpas/pas_ensure_heap_forced_into_reserved_memory.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_forced_into_reserved_memory):
* libpas/src/libpas/pas_ensure_heap_forced_into_reserved_memory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_ensure_heap_with_page_caches.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_with_page_caches):
* libpas/src/libpas/pas_ensure_heap_with_page_caches.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerable_page_malloc.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_enumerable_page_malloc_try_allocate_without_deallocating_padding):
* libpas/src/libpas/pas_enumerable_page_malloc.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerable_range_list.c: Added.
(pas_enumerable_range_list_append):
(pas_enumerable_range_list_iterate):
(pas_enumerable_range_list_iterate_remote):
* libpas/src/libpas/pas_enumerable_range_list.h: Added.
* libpas/src/libpas/pas_enumerate_bitfit_heaps.c: Added.
(view_callback):
(enumerate_bitfit_directory):
(enumerate_bitfit_heap_callback):
(pas_enumerate_bitfit_heaps):
* libpas/src/libpas/pas_enumerate_bitfit_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_initially_unaccounted_pages.c: Added.
(range_list_iterate_add_unaccounted_callback):
(range_list_iterate_exclude_accounted_callback):
(pas_enumerate_initially_unaccounted_pages):
* libpas/src/libpas/pas_enumerate_initially_unaccounted_pages.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_large_heaps.c: Added.
(range_list_iterate_add_large_payload_callback):
(record_span):
(large_map_hashtable_entry_callback):
(small_large_map_hashtable_entry_callback):
(tiny_large_map_second_level_hashtable_entry_callback):
(tiny_large_map_hashtable_entry_callback):
(pas_enumerate_large_heaps):
* libpas/src/libpas/pas_enumerate_large_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_segregated_heaps.c: Added.
(local_allocator_map_entry_create_empty):
(local_allocator_map_entry_create_deleted):
(local_allocator_map_entry_is_empty_or_deleted):
(local_allocator_map_entry_is_empty):
(local_allocator_map_entry_is_deleted):
(local_allocator_map_entry_get_key):
(local_allocator_map_key_get_hash):
(local_allocator_map_key_is_equal):
(for_each_view):
(collect_shared_page_directories_shared_page_directory_callback):
(collect_shared_page_directories_heap_callback):
(record_page_payload_and_meta):
(record_page_objects):
(enumerate_exclusive_view):
(enumerate_shared_view):
(enumerate_partial_view):
(shared_page_directory_view_callback):
(size_directory_view_callback):
(enumerate_segregated_heap_callback):
(consider_allocator):
(pas_enumerate_segregated_heaps):
* libpas/src/libpas/pas_enumerate_segregated_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_unaccounted_pages_as_meta.c: Added.
(pas_enumerate_unaccounted_pages_as_meta):
* libpas/src/libpas/pas_enumerate_unaccounted_pages_as_meta.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerator.c: Added.
(allocate):
(deallocate):
(pas_enumerator_create):
(pas_enumerator_destroy):
(pas_enumerator_allocate):
(pas_enumerator_read_compact):
(pas_enumerator_read):
(pas_enumerator_add_unaccounted_pages):
(pas_enumerator_exclude_accounted_page):
(pas_enumerator_exclude_accounted_pages):
(pas_enumerator_record):
(record_payload_span):
(pas_enumerator_record_page_payload_and_meta):
(pas_enumerator_for_each_heap):
(pas_enumerator_enumerate_all):
* libpas/src/libpas/pas_enumerator.h: Added.
(pas_enumerator_record_kind_get_string):
* libpas/src/libpas/pas_enumerator_internal.h: Added.
* libpas/src/libpas/pas_enumerator_region.c: Added.
(pas_enumerator_region_allocate):
(pas_enumerator_region_destroy):
* libpas/src/libpas/pas_enumerator_region.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_epoch.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_get_epoch):
* libpas/src/libpas/pas_epoch.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_exclusive_view_template_memo_table.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_exclusive_view_template_memo_table.h: Added.
(pas_exclusive_view_template_memo_key_create):
(pas_exclusive_view_template_memo_entry_create_empty):
(pas_exclusive_view_template_memo_entry_create_deleted):
(pas_exclusive_view_template_memo_entry_is_empty_or_deleted):
(pas_exclusive_view_template_memo_entry_is_empty):
(pas_exclusive_view_template_memo_entry_is_deleted):
(pas_exclusive_view_template_memo_entry_get_key):
(pas_exclusive_view_template_memo_key_get_hash):
(pas_exclusive_view_template_memo_key_is_equal):
* libpas/src/libpas/pas_extended_gcd.c: Added.
(pas_extended_gcd):
* libpas/src/libpas/pas_extended_gcd.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_fast_large_free_heap.c: Added.
(key_compare_callback):
(get_x_key_callback):
(get_y_key_callback):
(initialize_cartesian_config):
(pas_fast_large_free_heap_construct):
(insert_node):
(remove_node):
(dump_heap):
(fast_find_first):
(fast_find_by_end):
(fast_read_cursor):
(fast_write_cursor):
(fast_merge):
(fast_remove):
(fast_append):
(fast_commit):
(fast_dump):
(fast_add_mapped_bytes):
(initialize_generic_heap_config):
(pas_fast_large_free_heap_try_allocate):
(pas_fast_large_free_heap_deallocate):
(pas_fast_large_free_heap_for_each_free):
(pas_fast_large_free_heap_get_num_free_bytes):
(pas_fast_large_free_heap_validate):
(pas_fast_large_free_heap_dump_to_printf):
* libpas/src/libpas/pas_fast_large_free_heap.h: Added.
(pas_fast_large_free_heap_get_num_mapped_bytes):
* libpas/src/libpas/pas_fast_megapage_cache.c: Added.
(table_set_by_index):
(pas_fast_megapage_cache_try_allocate):
* libpas/src/libpas/pas_fast_megapage_cache.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_fast_megapage_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_fast_megapage_kind_get_string):
* libpas/src/libpas/pas_fast_megapage_table.c: Added.
(pas_fast_megapage_table_initialize_static):
(pas_fast_megapage_table_set_by_index):
* libpas/src/libpas/pas_fast_megapage_table.h: Added.
(pas_fast_megapage_table_get_by_index):
(pas_fast_megapage_table_get):
* libpas/src/libpas/pas_fast_path_allocation_result.h: Added.
(pas_fast_path_allocation_result_create):
(pas_fast_path_allocation_result_create_need_slow):
(pas_fast_path_allocation_result_create_out_of_memory):
(pas_fast_path_allocation_result_create_success):
(pas_fast_path_allocation_result_from_allocation_result):
(pas_fast_path_allocation_result_to_allocation_result):
* libpas/src/libpas/pas_fast_path_allocation_result_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_fast_path_allocation_result_kind_get_string):
* libpas/src/libpas/pas_fast_tls.h: Added.
* libpas/src/libpas/pas_fd_stream.c: Copied from Source/WTF/wtf/FastTLS.h.
(fd_stream_vprintf):
(pas_fd_stream_vprintf):
* libpas/src/libpas/pas_fd_stream.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_first_level_tiny_large_map_entry.h: Added.
(pas_first_level_tiny_large_map_entry_create_empty):
(pas_first_level_tiny_large_map_entry_create_deleted):
(pas_first_level_tiny_large_map_entry_is_empty_or_deleted):
(pas_first_level_tiny_large_map_entry_is_empty):
(pas_first_level_tiny_large_map_entry_is_deleted):
(pas_first_level_tiny_large_map_entry_get_key):
(pas_first_level_tiny_large_map_key_get_hash):
(pas_first_level_tiny_large_map_key_is_equal):
* libpas/src/libpas/pas_found_bit_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_found_bit_index_create_empty):
(pas_found_bit_index_create):
* libpas/src/libpas/pas_found_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_found_index_create_empty):
(pas_found_index_create):
(pas_found_index_create_unsuccessful):
* libpas/src/libpas/pas_free_granules.c: Added.
(pas_free_granules_compute_and_mark_decommitted):
(pas_free_granules_unmark_decommitted):
(pas_free_granules_decommit_after_locking_range):
* libpas/src/libpas/pas_free_granules.h: Added.
(pas_free_granules_is_free):
* libpas/src/libpas/pas_free_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_free_range_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_free_range_kind_get_string):
* libpas/src/libpas/pas_full_alloc_bits.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_full_alloc_bits_create_empty):
(pas_full_alloc_bits_create):
* libpas/src/libpas/pas_full_alloc_bits_inlines.h: Added.
(pas_full_alloc_bits_create_for_exclusive):
(pas_full_alloc_bits_create_for_partial):
(pas_full_alloc_bits_create_for_view_and_directory):
(pas_full_alloc_bits_create_for_view):
* libpas/src/libpas/pas_generic_large_free_heap.h: Added.
(pas_generic_large_free_heap_merge_physical):
(pas_generic_large_free_heap_try_allocate_test_allocation_candidate):
(pas_generic_large_free_heap_try_allocate):
* libpas/src/libpas/pas_get_allocation_size.h: Added.
(pas_get_allocation_size):
* libpas/src/libpas/pas_get_heap.h: Added.
(pas_get_heap_known_segregated):
(pas_get_heap):
* libpas/src/libpas/pas_get_object_kind.h: Added.
(pas_get_object_kind):
* libpas/src/libpas/pas_get_page_base.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_get_page_base):
* libpas/src/libpas/pas_has_object.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_has_object):
* libpas/src/libpas/pas_hashtable.h: Added.
* libpas/src/libpas/pas_heap.c: Added.
(pas_heap_create):
(pas_heap_get_type_size):
(pas_heap_get_num_free_bytes):
(for_each_live_object_small_object_callback):
(for_each_live_object_large_object_callback):
(pas_heap_for_each_live_object):
(pas_heap_compute_summary):
(pas_heap_reset_heap_ref):
(pas_heap_ensure_size_directory_for_count_slow):
* libpas/src/libpas/pas_heap.h: Added.
(pas_heap_for_large_heap):
(pas_heap_for_segregated_heap):
* libpas/src/libpas/pas_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_config_activate):
* libpas/src/libpas/pas_heap_config.h: Added.
(pas_heap_config_segregated_page_config_ptr_for_variant):
(pas_heap_config_bitfit_page_config_ptr_for_variant):
(pas_heap_config_segregated_page_config_for_variant):
(pas_heap_config_bitfit_page_config_for_variant):
* libpas/src/libpas/pas_heap_config_inlines.h: Added.
* libpas/src/libpas/pas_heap_config_kind.c: Added.
(pas_heap_config_kind_for_each):
* libpas/src/libpas/pas_heap_config_kind.def: Added.
* libpas/src/libpas/pas_heap_config_kind.h: Added.
(pas_heap_config_kind_get_string):
(pas_heap_config_kind_get_config):
(pas_heap_config_kind_is_active):
* libpas/src/libpas/pas_heap_config_utils.c: Added.
(pas_heap_config_utils_null_activate):
(pas_heap_config_utils_for_each_shared_page_directory):
(pas_heap_config_utils_for_each_shared_page_directory_remote):
(pas_heap_config_utils_allocate_aligned):
(pas_heap_config_utils_prepare_to_enumerate):
* libpas/src/libpas/pas_heap_config_utils.h: Added.
* libpas/src/libpas/pas_heap_config_utils_inlines.h: Added.
* libpas/src/libpas/pas_heap_for_config.c: Added.
(pas_heap_for_config_allocate):
(pas_heap_for_page_config_kind_allocate):
(pas_heap_for_page_config_allocate):
(pas_heap_for_config_allocate_with_alignment):
(pas_heap_for_page_config_allocate_with_alignment):
(pas_heap_for_config_allocate_with_manual_alignment):
(pas_heap_for_page_config_kind_allocate_with_manual_alignment):
(pas_heap_for_page_config_allocate_with_manual_alignment):
(pas_heap_for_config_deallocate):
(pas_heap_for_page_config_kind_deallocate):
(pas_heap_for_page_config_deallocate):
* libpas/src/libpas/pas_heap_for_config.h: Added.
* libpas/src/libpas/pas_heap_inlines.h: Added.
(pas_heap_ensure_size_directory_for_count):
* libpas/src/libpas/pas_heap_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_kind_get_string):
* libpas/src/libpas/pas_heap_lock.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_lock.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_page_provider.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_ref.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_slow):
* libpas/src/libpas/pas_heap_ref.h: Added.
(pas_ensure_heap):
* libpas/src/libpas/pas_heap_ref_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_ref_kind_get_string):
* libpas/src/libpas/pas_heap_ref_prefix.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_runtime_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_summary.c: Added.
(pas_heap_summary_validate):
(pas_heap_summary_dump):
* libpas/src/libpas/pas_heap_summary.h: Added.
(pas_heap_summary_create_empty):
(pas_heap_summary_is_empty):
(pas_heap_summary_add):
(pas_heap_summary_committed_objects):
(pas_heap_summary_total):
(pas_heap_summary_fragmentation):
* libpas/src/libpas/pas_heap_table.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_table_try_allocate_index):
* libpas/src/libpas/pas_heap_table.h: Added.
(pas_heap_table_has_index):
(pas_heap_table_get_index):
* libpas/src/libpas/pas_heap_table_state.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_table_state_get_string):
* libpas/src/libpas/pas_immortal_heap.c: Added.
(bump_is_ok):
(pas_immortal_heap_allocate_with_manual_alignment):
(pas_immortal_heap_allocate_with_alignment):
(pas_immortal_heap_allocate):
(pas_immortal_heap_hold_lock_and_allocate):
(pas_immortal_heap_allocate_with_heap_lock_hold_mode):
(pas_immortal_heap_allocate_with_alignment_and_heap_lock_hold_mode):
* libpas/src/libpas/pas_immortal_heap.h: Added.
* libpas/src/libpas/pas_immutable_vector.h: Added.
* libpas/src/libpas/pas_internal_config.h: Added.
* libpas/src/libpas/pas_intrinsic_allocation_result.h: Added.
(pas_intrinsic_allocation_result_create_empty):
(pas_intrinsic_allocation_result_create):
(pas_intrinsic_allocation_result_identity):
(pas_intrinsic_allocation_result_zero):
(pas_intrinsic_allocation_result_set_errno):
(pas_intrinsic_allocation_result_crash_on_error):
* libpas/src/libpas/pas_intrinsic_heap_support.h: Added.
* libpas/src/libpas/pas_large_free.h: Added.
(pas_large_free_create_empty):
(pas_large_free_is_empty):
(pas_large_free_size):
(pas_large_free_offset_in_type_at_end):
* libpas/src/libpas/pas_large_free_heap_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_free_heap_declarations.def: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_free_heap_deferred_commit_log.c: Added.
(pas_large_free_heap_deferred_commit_log_construct):
(pas_large_free_heap_deferred_commit_log_destruct):
(pas_large_free_heap_deferred_commit_log_add):
(dump_large_commit):
(commit):
(commit_all):
(pas_large_free_heap_deferred_commit_log_commit_all):
(pas_large_free_heap_deferred_commit_log_pretend_to_commit_all):
* libpas/src/libpas/pas_large_free_heap_deferred_commit_log.h: Added.
* libpas/src/libpas/pas_large_free_heap_definitions.def: Added.
* libpas/src/libpas/pas_large_free_heap_helpers.c: Added.
(large_utility_aligned_allocator):
(initialize_config):
(pas_large_free_heap_helpers_try_allocate_with_alignment):
(pas_large_free_heap_helpers_deallocate):
(pas_large_free_heap_helpers_compute_summary):
* libpas/src/libpas/pas_large_free_heap_helpers.h: Added.
* libpas/src/libpas/pas_large_free_inlines.h: Added.
(pas_large_free_create_merged):
(pas_large_free_split):
(pas_large_free_allocation_candidate):
(pas_large_free_usable_space):
(pas_large_allocation_result_create_empty):
(pas_large_free_allocate):
(pas_large_free_create_merged_for_sure):
(pas_large_free_can_merge):
(pas_large_allocation_result_as_allocation_result):
* libpas/src/libpas/pas_large_free_visitor.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_heap.c: Added.
(pas_large_heap_construct):
(aligned_allocator):
(initialize_config):
(pas_large_heap_try_allocate):
(pas_large_heap_try_deallocate):
(pas_large_heap_try_shrink):
(pas_large_heap_shove_into_free):
(for_each_live_object_entry_callback):
(pas_large_heap_for_each_live_object):
(pas_large_heap_for_object):
(pas_large_heap_get_num_free_bytes):
(compute_summary_live_object_callback):
(pas_large_heap_compute_summary):
* libpas/src/libpas/pas_large_heap.h: Added.
* libpas/src/libpas/pas_large_heap_physical_page_sharing_cache.c: Added.
(large_aligned_allocator):
(pas_large_heap_physical_page_sharing_cache_construct):
(pas_large_heap_physical_page_sharing_cache_try_allocate_with_alignment):
* libpas/src/libpas/pas_large_heap_physical_page_sharing_cache.h: Added.
* libpas/src/libpas/pas_large_map.c: Added.
(pas_large_map_find):
(pas_large_map_add):
(pas_large_map_take):
(pas_large_map_for_each_entry):
* libpas/src/libpas/pas_large_map.h: Added.
* libpas/src/libpas/pas_large_map_entry.h: Added.
(pas_large_map_entry_create_empty):
(pas_large_map_entry_create_deleted):
(pas_large_map_entry_is_empty_or_deleted):
(pas_large_map_entry_is_empty):
(pas_large_map_entry_is_deleted):
(pas_large_map_entry_get_key):
(pas_large_map_key_get_hash):
(pas_large_map_key_is_equal):
* libpas/src/libpas/pas_large_sharing_pool.c: Added.
(node_compare_callback):
(inner_key_compare_callback):
(update_min_epoch):
(validate_min_heap):
(validate_node):
(validate_node_if_asserting_aggressively):
(create_node):
(create_and_insert):
(boot_tree):
(destroy_node):
(remove_from_min_heap):
(remove_and_destroy):
(predecessor):
(successor):
(states_match):
(is_eligible):
(belongs_in_min_heap):
(update_min_heap):
(split_node_and_get_right_impl):
(split_node_and_get_right):
(split_node_and_get_left):
(merge_if_possible):
(node_containing):
(min_node_for_range):
(max_node_for_range):
(splat_live_bytes):
(should_do_commit_stuff_to):
(splat_command_get_string):
(splat_command_get_free_mode):
(dump_large_commit):
(try_splat_impl):
(try_splat):
(splat):
(pas_large_sharing_pool_boot_free):
(pas_large_sharing_pool_free):
(pas_large_sharing_pool_allocate_and_commit):
(pas_large_sharing_pool_decommit_least_recently_used):
(pas_large_sharing_pool_validate):
(pas_large_sharing_pool_compute_summary):
(pas_large_sharing_pool_for_each):
* libpas/src/libpas/pas_large_sharing_pool.h: Added.
(pas_large_sharing_node_heap_compare):
(pas_large_sharing_node_heap_get_index):
(pas_large_sharing_node_heap_set_index):
* libpas/src/libpas/pas_large_sharing_pool_epoch_update_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_large_sharing_pool_epoch_update_mode_get_string):
* libpas/src/libpas/pas_large_utility_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_utility_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_line_word_config.h: Added.
(pas_line_word_config_count_low_zeroes_8_bit):
(pas_line_word_config_c…

Source/JavaScriptCore/CMakeLists.txt

@@ -760,6 +760,7 @@ set(JavaScriptCore_PRIVATE_FRAMEWORK_HEADERS
     jit/AssemblyHelpersSpoolers.h
     jit/CCallHelpers.h
     jit/ExecutableAllocator.h
+    jit/ExecutableMemoryHandle.h
     jit/FPRInfo.h
     jit/GCAwareJITStubRoutine.h
     jit/GPRInfo.h

Source/JavaScriptCore/ChangeLog

@@ -1,3 +1,46 @@
+2021-07-12  Filip Pizlo  <fpizlo@apple.com> and Yusuke Suzuki  <ysuzuki@apple.com>
+
+        New malloc algorithm
+        https://bugs.webkit.org/show_bug.cgi?id=226404
+
+        Reviewed by Yusuke Suzuki.
+
+        Switch the ExecutableAllocator to the libpas jit_heap. The libpas jit_heap uses two size
+        categories of bitfit and a large heap, and is happy to do its approximate first-fit in any
+        ranges of memory you give it. Jit_heap never allocates metadata inside the memory it manages.
+        Allocations and deallocations take a constant-bounded amount of time except for unusual
+        situations, and are protected by fine-grained locking in most cases. Decommit follows libpas
+        scavenging policy. This alone is a speed-up on Speedometer (probably about 1% or more).
+
+        Also expose some libpas introspection via $vm.
+
+        * CMakeLists.txt:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/MacroAssemblerCodeRef.h:
+        * jit/ExecutableAllocator.cpp:
+        (JSC::ExecutableAllocator::underMemoryPressure):
+        (JSC::ExecutableAllocator::memoryPressureMultiplier):
+        (JSC::ExecutableAllocator::allocate):
+        (JSC::ExecutableAllocator::committedByteCount):
+        (JSC::ExecutableMemoryHandle::createImpl):
+        (JSC::ExecutableMemoryHandle::~ExecutableMemoryHandle):
+        (JSC::ExecutableMemoryHandle::sizeInBytes const):
+        (JSC::ExecutableMemoryHandle::shrink):
+        * jit/ExecutableAllocator.h:
+        * jit/ExecutableMemoryHandle.h: Added.
+        (JSC::ExecutableMemoryHandle::start const):
+        (JSC::ExecutableMemoryHandle::end const):
+        (JSC::ExecutableMemoryHandle::startAsInteger const):
+        (JSC::ExecutableMemoryHandle::endAsInteger const):
+        (JSC::ExecutableMemoryHandle::containsIntegerAddress const):
+        (JSC::ExecutableMemoryHandle::contains const):
+        (JSC::ExecutableMemoryHandle::key const):
+        (JSC::ExecutableMemoryHandle::dump const):
+        (JSC::ExecutableMemoryHandle::ExecutableMemoryHandle):
+        * tools/JSDollarVM.cpp:
+        (JSC::JSC_DEFINE_HOST_FUNCTION):
+        (JSC::JSDollarVM::finishCreation):
+
 2021-07-12  Mark Lam  <mark.lam@apple.com>
 
         Revert r277027: breaks GC.

Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -349,6 +349,7 @@
 		0F4F29E018B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */; };
 		0F4F828C1E31B9760075184C /* StochasticSpaceTimeMutatorScheduler.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4F828A1E31B9710075184C /* StochasticSpaceTimeMutatorScheduler.h */; };
 		0F50AF3C193E8B3900674EE8 /* DFGStructureClobberState.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */; };
+		0F5193F7266C432D00483A2C /* ExecutableMemoryHandle.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5193F6266C432D00483A2C /* ExecutableMemoryHandle.h */; settings = {ATTRIBUTES = (Private, ); }; };
 		0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5513A51D5A682A00C32BD8 /* FreeList.h */; settings = {ATTRIBUTES = (Private, ); }; };
 		0F55989817C86C5800A1E543 /* ToNativeFromValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F55989717C86C5600A1E543 /* ToNativeFromValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
 		0F55F0F514D1063C00AC7649 /* AbstractPC.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F55F0F214D1063600AC7649 /* AbstractPC.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -2672,6 +2673,7 @@
 		0F4F82891E31B9710075184C /* StochasticSpaceTimeMutatorScheduler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StochasticSpaceTimeMutatorScheduler.cpp; sourceTree = "<group>"; };
 		0F4F828A1E31B9710075184C /* StochasticSpaceTimeMutatorScheduler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StochasticSpaceTimeMutatorScheduler.h; sourceTree = "<group>"; };
 		0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStructureClobberState.h; path = dfg/DFGStructureClobberState.h; sourceTree = "<group>"; };
+		0F5193F6266C432D00483A2C /* ExecutableMemoryHandle.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ExecutableMemoryHandle.h; sourceTree = "<group>"; };
 		0F5513A51D5A682A00C32BD8 /* FreeList.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FreeList.h; sourceTree = "<group>"; };
 		0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FreeList.cpp; sourceTree = "<group>"; };
 		0F55989717C86C5600A1E543 /* ToNativeFromValue.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ToNativeFromValue.h; sourceTree = "<group>"; };
@@ -6248,6 +6250,7 @@
 				0FF054F81AC35B4400E5BE57 /* ExecutableAllocationFuzz.h */,
 				A7B48DB60EE74CFC00DCBDB6 /* ExecutableAllocator.cpp */,
 				A7B48DB50EE74CFC00DCBDB6 /* ExecutableAllocator.h */,
+				0F5193F6266C432D00483A2C /* ExecutableMemoryHandle.h */,
 				0F24E53E17EA9F5900ABB217 /* FPRInfo.h */,
 				0F766D2D15A8DCDD008F363E /* GCAwareJITStubRoutine.cpp */,
 				0F766D2E15A8DCDD008F363E /* GCAwareJITStubRoutine.h */,
@@ -9800,6 +9803,7 @@
 				147341CC1DC02D7200AA29BA /* ExecutableBase.h in Headers */,
 				E35A0B9D220AD87A00AC4474 /* ExecutableBaseInlines.h in Headers */,
 				14142E531B796EDD00F4BF4B /* ExecutableInfo.h in Headers */,
+				0F5193F7266C432D00483A2C /* ExecutableMemoryHandle.h in Headers */,
 				0F60FE901FFC37020003320A /* ExecutableToCodeBlockEdge.h in Headers */,
 				0F56A1D315000F35002992B1 /* ExecutionCounter.h in Headers */,
 				0F44A7B020BF68620022B171 /* ExitFlag.h in Headers */,

Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h

@@ -25,9 +25,9 @@
 
 #pragma once
 
+#include "ExecutableMemoryHandle.h"
 #include "JSCPtrTag.h"
 #include <wtf/DataLog.h>
-#include <wtf/MetaAllocatorHandle.h>
 #include <wtf/PrintStream.h>
 #include <wtf/RefPtr.h>
 #include <wtf/text/CString.h>
@@ -54,7 +54,6 @@
 
 namespace JSC {
 
-typedef WTF::MetaAllocatorHandle ExecutableMemoryHandle;
 template<PtrTag> class MacroAssemblerCodePtr;
 
 enum OpcodeID : unsigned;

Source/JavaScriptCore/jit/ExecutableAllocator.cpp

@@ -33,13 +33,19 @@
 #include "LinkBuffer.h"
 #include <wtf/FastBitVector.h>
 #include <wtf/FileSystem.h>
-#include <wtf/MetaAllocator.h>
 #include <wtf/PageReservation.h>
 #include <wtf/ProcessID.h>
+#include <wtf/RedBlackTree.h>
 #include <wtf/Scope.h>
 #include <wtf/SystemTracing.h>
 #include <wtf/WorkQueue.h>
 
+#if USE(LIBPAS_JIT_HEAP)
+#include <bmalloc/jit_heap.h>
+#else
+#include <wtf/MetaAllocator.h>
+#endif
+
 #if HAVE(IOS_JIT_RESTRICTIONS)
 #include <wtf/cocoa/Entitlements.h>
 #endif
@@ -448,7 +454,9 @@ class FixedVMPoolExecutableAllocator final {
             }
 #else
             m_allocator.addFreshFreeSpace(reservation.base, reservation.size);
+#if !USE(LIBPAS_JIT_HEAP)
             ASSERT(bytesReserved() == reservation.size); // Since our executable memory is fixed-sized, bytesReserved is never changed after initialization.
+#endif
 #endif
         }
     }
@@ -465,7 +473,9 @@ class FixedVMPoolExecutableAllocator final {
 
     RefPtr<ExecutableMemoryHandle> allocate(size_t sizeInBytes)
     {
-#if ENABLE(JUMP_ISLANDS)
+#if USE(LIBPAS_JIT_HEAP)
+        return ExecutableMemoryHandle::createImpl(sizeInBytes);
+#elif ENABLE(JUMP_ISLANDS)
         Locker locker { getLock() };
 
         unsigned start = 0;
@@ -489,6 +499,7 @@ class FixedVMPoolExecutableAllocator final {
 
     Lock& getLock() WTF_RETURNS_LOCK(m_lock) { return m_lock; }
 
+#if !USE(LIBPAS_JIT_HEAP)
     // Non atomic
     size_t bytesAllocated() 
     {
@@ -514,6 +525,7 @@ class FixedVMPoolExecutableAllocator final {
         });
         return result;
     }
+#endif
 
     bool isInAllocatedMemory(const AbstractLocker& locker, void* address)
     {
@@ -535,6 +547,7 @@ class FixedVMPoolExecutableAllocator final {
     }
 #endif
 
+#if !USE(LIBPAS_JIT_HEAP)
     MetaAllocator::Statistics currentStatistics()
     {
         Locker locker { getLock() };
@@ -547,9 +560,25 @@ class FixedVMPoolExecutableAllocator final {
         });
         return result;
     }
+#endif // !USE(LIBPAS_JIT_HEAP)
+
+#if USE(LIBPAS_JIT_HEAP)
+    void handleWillBeReleased(ExecutableMemoryHandle& handle)
+    {
+#if ENABLE(JUMP_ISLANDS)
+        if (m_islandsForJumpSourceLocation.isEmpty())
+            return;
+
+        Locker locker { getLock() };
+        handleWillBeReleased(locker, handle);
+#else // ENABLE(JUMP_ISLANDS) -> so !ENABLE(JUMP_ISLANDS)
+        UNUSED_PARAM(handle);
+#endif // ENABLE(JUMP_ISLANDS) -> so end of !ENABLE(JUMP_ISLANDS)
+    }
+#endif // USE(LIBPAS_JIT_HEAP)
 
 #if ENABLE(JUMP_ISLANDS)
-    void handleWillBeReleased(const LockHolder& locker, MetaAllocatorHandle& handle)
+    void handleWillBeReleased(const LockHolder& locker, ExecutableMemoryHandle& handle)
     {
         if (m_islandsForJumpSourceLocation.isEmpty())
             return;
@@ -685,15 +714,44 @@ class FixedVMPoolExecutableAllocator final {
 #endif // ENABLE(JUMP_ISLANDS)
 
 private:
-    class Allocator : public MetaAllocator {
+    class Allocator
+#if !USE(LIBPAS_JIT_HEAP)
+        : public MetaAllocator
+#endif
+    {
+#if !USE(LIBPAS_JIT_HEAP)
         using Base = MetaAllocator;
+#endif
     public:
         Allocator(FixedVMPoolExecutableAllocator& allocator)
+#if !USE(LIBPAS_JIT_HEAP)
             : Base(allocator.getLock(), jitAllocationGranule, pageSize()) // round up all allocations to 32 bytes
-            , m_fixedAllocator(allocator)
+            ,
+#else
+            :
+#endif
+            m_fixedAllocator(allocator)
         {
         }
 
+#if USE(LIBPAS_JIT_HEAP)
+        void addFreshFreeSpace(void* start, size_t sizeInBytes)
+        {
+            RELEASE_ASSERT(!m_start);
+            RELEASE_ASSERT(!m_end);
+            m_start = reinterpret_cast<uintptr_t>(start);
+            m_end = m_start + sizeInBytes; 
+            jit_heap_add_fresh_memory(pas_range_create(m_start, m_end));
+        }
+
+        bool isInAllocatedMemory(const AbstractLocker&, void* address)
+        {
+            uintptr_t addressAsInt = reinterpret_cast<uintptr_t>(address);
+            return addressAsInt >= m_start && addressAsInt < m_end;
+        }
+#endif // USE(LIBPAS_JIT_HEAP)
+
+#if !USE(LIBPAS_JIT_HEAP)
         FreeSpacePtr allocateNewSpace(size_t&) override
         {
             // We're operating in a fixed pool, so new allocation is always prohibited.
@@ -709,8 +767,13 @@ class FixedVMPoolExecutableAllocator final {
         {
             m_fixedAllocator.m_reservation.decommit(page, pageSize() * count);
         }
+#endif // !USE(LIBPAS_JIT_HEAP)
 
         FixedVMPoolExecutableAllocator& m_fixedAllocator;
+#if USE(LIBPAS_JIT_HEAP)
+        uintptr_t m_start { 0 };
+        uintptr_t m_end { 0 };
+#endif // USE(LIBPAS_JIT_HEAP)
     };
 
 #if ENABLE(JUMP_ISLANDS)
@@ -748,11 +811,13 @@ class FixedVMPoolExecutableAllocator final {
             return islandsPerPage;
         }
 
+#if !USE(LIBPAS_JIT_HEAP)
         void release(const LockHolder& locker, MetaAllocatorHandle& handle) final
         {
             m_fixedAllocator.handleWillBeReleased(locker, handle);
             Base::release(locker, handle);
         }
+#endif
 
         void* allocateIsland()
         {
@@ -880,14 +945,21 @@ bool ExecutableAllocator::isValid() const
 
 bool ExecutableAllocator::underMemoryPressure()
 {
+#if USE(LIBPAS_JIT_HEAP)
+    return Base::underMemoryPressure();
+#else // USE(LIBPAS_JIT_HEAP) -> so start of !USE(LIBPAS_JIT_HEAP)
     FixedVMPoolExecutableAllocator* allocator = g_jscConfig.fixedVMPoolExecutableAllocator;
     if (!allocator)
         return Base::underMemoryPressure();
     return allocator->bytesAllocated() > allocator->bytesReserved() / 2;
+#endif // USE(LIBPAS_JIT_HEAP) -> so end of !USE(LIBPAS_JIT_HEAP)
 }
 
 double ExecutableAllocator::memoryPressureMultiplier(size_t addedMemoryUsage)
 {
+#if USE(LIBPAS_JIT_HEAP)
+    return Base::memoryPressureMultiplier(addedMemoryUsage);
+#else // USE(LIBPAS_JIT_HEAP) -> so start of !USE(LIBPAS_JIT_HEAP)
     FixedVMPoolExecutableAllocator* allocator = g_jscConfig.fixedVMPoolExecutableAllocator;
     if (!allocator)
         return Base::memoryPressureMultiplier(addedMemoryUsage);
@@ -904,17 +976,20 @@ double ExecutableAllocator::memoryPressureMultiplier(size_t addedMemoryUsage)
     if (result < 1.0)
         result = 1.0;
     return result;
+#endif // USE(LIBPAS_JIT_HEAP) -> so end of !USE(LIBPAS_JIT_HEAP)
 }
 
 RefPtr<ExecutableMemoryHandle> ExecutableAllocator::allocate(size_t sizeInBytes, JITCompilationEffort effort)
 {
     FixedVMPoolExecutableAllocator* allocator = g_jscConfig.fixedVMPoolExecutableAllocator;
     if (!allocator)
         return Base::allocate(sizeInBytes, effort);
+#if !USE(LIBPAS_JIT_HEAP)
     if (Options::logExecutableAllocation()) {
         MetaAllocator::Statistics stats = allocator->currentStatistics();
         dataLog("Allocating ", sizeInBytes, " bytes of executable memory with ", stats.bytesAllocated, " bytes allocated, ", stats.bytesReserved, " bytes reserved, and ", stats.bytesCommitted, " committed.\n");
     }
+#endif
 
     if (effort != JITCompilationCanFail && Options::reportMustSucceedExecutableAllocations()) {
         dataLog("Allocating ", sizeInBytes, " bytes of executable memory with JITCompilationMustSucceed.\n");
@@ -925,6 +1000,7 @@ RefPtr<ExecutableMemoryHandle> ExecutableAllocator::allocate(size_t sizeInBytes,
         && doExecutableAllocationFuzzingIfEnabled() == PretendToFailExecutableAllocation)
         return nullptr;
 
+#if !USE(LIBPAS_JIT_HEAP)
     if (effort == JITCompilationCanFail) {
         // Don't allow allocations if we are down to reserve.
         size_t bytesAllocated = allocator->bytesAllocated() + sizeInBytes;
@@ -936,6 +1012,7 @@ RefPtr<ExecutableMemoryHandle> ExecutableAllocator::allocate(size_t sizeInBytes,
             return nullptr;
         }
     }
+#endif // !USE(LIBPAS_JIT_HEAP)
 
     RefPtr<ExecutableMemoryHandle> result = allocator->allocate(sizeInBytes);
     if (!result) {
@@ -973,10 +1050,14 @@ Lock& ExecutableAllocator::getLock() const
 
 size_t ExecutableAllocator::committedByteCount()
 {
+#if USE(LIBPAS_JIT_HEAP)
+    return Base::committedByteCount();
+#else // USE(LIBPAS_JIT_HEAP) -> so start of !USE(LIBPAS_JIT_HEAP)
     FixedVMPoolExecutableAllocator* allocator = g_jscConfig.fixedVMPoolExecutableAllocator;
     if (!allocator)
         return Base::committedByteCount();
     return allocator->bytesCommitted();
+#endif // USE(LIBPAS_JIT_HEAP) -> so end of !USE(LIBPAS_JIT_HEAP)
 }
 
 #if ENABLE(META_ALLOCATOR_PROFILE)
@@ -1110,6 +1191,33 @@ void dumpJITMemory(const void* dst, const void* src, size_t size)
 #endif
 }
 
+#if USE(LIBPAS_JIT_HEAP)
+RefPtr<ExecutableMemoryHandle> ExecutableMemoryHandle::createImpl(size_t sizeInBytes)
+{
+    void* key = jit_heap_try_allocate(sizeInBytes);
+    if (!key)
+        return nullptr;
+    return adoptRef(new ExecutableMemoryHandle(MemoryPtr::makeFromRawPointer(key)));
+}
+
+ExecutableMemoryHandle::~ExecutableMemoryHandle()
+{
+    FixedVMPoolExecutableAllocator* allocator = g_jscConfig.fixedVMPoolExecutableAllocator;
+    allocator->handleWillBeReleased(*this);
+    jit_heap_deallocate(key());
+}
+
+size_t ExecutableMemoryHandle::sizeInBytes() const
+{
+    return jit_heap_get_size(key());
+}
+
+void ExecutableMemoryHandle::shrink(size_t newSizeInBytes)
+{
+    jit_heap_shrink(key(), newSizeInBytes);
+}
+#endif
+
 } // namespace JSC
 
 #endif // ENABLE(JIT)