Skip to content

v2: NUMA, PThreads, Sleeping, & Naming

Choose a tag to compare

View all tags
@ashvardanian ashvardanian released this 02 Jul 15:13
v2.0.0
fd6d2bf

Much of the new functionality would require tuning, but the ground work for NUMA-aware efficient simultaneous multithreading in C++, C, and Rust on Linux is here 🥳

Breaking Changes

The default thread-pool class now splits its broadcasting logic into the actual "broadcast" and a "join". It also adds support for "exclusive spawns" when the current thread is not collaborating with the $N$ workers on a given task. It allows more fine-grained control, with one thread simultaneously participating in one pool and managing others.

Moreover, now the library supports "mood swings". By default, it's ready to grind_k. However, if you want to consume fewer resources and don't anticipate any additional work for a few milliseconds, consider marking it as lazy_k. It will negatively affect the latency of your next call.

NUMA Support On Linux

On Linux, several new core classes are available, powered by two optional dependencies: libnuma and pthreads. The first helps us relate colocated CPU cores and their associated memory banks, and the latter facilitates pinning, naming, and scheduling logical execution threads.

  • linux_numa_allocator - a stateful allocator tied to a particular NUMA node,
  • linux_colocated_pool - a thread pool with all threads running on the same NUMA node,
  • linux_distributed_pool - a thread-pool spanning & balancing many colocated pools.

The allocator can and should be used with the new C++23-like but backward-compatible allocate_at_least interface:

fu::linux_numa_allocator<int> allocator(1, 1024*1024*1024); // node_id=1, page_size=1GB
auto [ptr, count] = allocator.allocate_at_least(256*1024*1024); // 256M = 1GB of integers

Huge Pages Functionality

To further reduce the memory access latency for user-space applications, one can update the "Huge Page" settings of the Operating System for a more coarse-grained virtual address range allocation. Those "virtual" addresses need to be mapped to "physical" ones, and to accelerate operations, the CPU maintains a small cache known as the Translation Lookaside Buffer (TLB). The fewer addresses it needs to map, the more likely it is to stay relevant, and global memory accesses - are avoided.

Linux typically supports optional 2 MB and 1 GB pages (512 times and ~260,000 times larger than the standard 4 KB pages), thus allowing us to reduce the TLB load by 5 orders of magnitude. It even comes with a HugeTLBfs library for querying some of those settings, but linking to it isn't trivial, and the functionality is minimal. So instead, Fork Union now directly queries the /sysfs and allows overwriting it if the application is granted sufficient permissions.

New Function on x86, Arm, and RISC-V

Beyond the basic std::this_thread::yield(), the library now leverages architecture-specific instructions for more efficient busy-waiting:

  • x86_pause_t - uses the PAUSE instruction for better SMT performance during spin-waits.
  • x86_tpause_t - employs the newer TPAUSE instruction for timed microsecond-precision waits, transitioning the CPU into the C0.2 power-saving state.
  • arm64_yield_t - utilizes the YIELD hint instruction for better power efficiency.
  • arm64_wfet_t - leverages "Wait For Event Timed" WFET for event-driven sleeping with precise timeout control.
  • risc5_pause_t - uses the Zihintpause extension's PAUSE instruction for efficient spin-loop hints.

Similar to SimSIMD and StringZilla, the library precompiles all relevant binaries and selects the best-supported variant at runtime. You can query supported features and log them with the provided functionality in C++:

fu::capabilities_t cpu_caps = fu::cpu_capabilities(); // Depends mostly on the CPU model
fu::capabilities_t ram_caps = fu::ram_capabilities(); // Depends mostly on the OS settings
bool has_waitpkg = cpu_caps & fu::capability_x86_tpause_k; // Check for something specific
fu::log_capabilities_t {}(cpu_caps | ram_caps, colors); // Print to console
fu::log_numa_topology_t {}(numa_topology, colors); // Print the entire topology

New C Library

To avoid reimplementing all this logic in every programming language, a stable C API for core functionality is provided, covering both safe and unsafe operations — perfect for creating higher-level wrappers and parallel runtimes for other programming languages, such as Rust, Python, or Java.

Major

  • Break: Drop unsafe_for_n* APIs (3a168b3)
  • Break: Exclusive spawn, unsafe APIs (287502f)

Minor

  • Add: named_spawn for Rust (61cecb5)
  • Add: Missing metadata methods in Rust (951e216)
  • Add: unique_padded_buffer to manage padded allocations (b93ffff)
  • Add: Capabilities logging (afe96f5)
  • Add: Shared library logic (4a1608f)
  • Add: APIs to check the library version (e0e156c)
  • Add: Logging CPU details (b976cab)
  • Add: ram_page_settings::try_change (9d8c6e5)
  • Add: Store numa_node::socket_id (ebdd4b6)
  • Add: Fetching cpu_capabilities() (76c5618)
  • Add: WFET variant for Arm64 (c65a9b9)
  • Add: Draft C library (ac19713)
  • Add: Tests for helpers (04f98f8)
  • Add: Co-prime permutation ranges (a5be78f)
  • Add: numa_topology move & copy-like funcs (89556c3)
  • Add: invoke_distributed_for_slices draft (d659280)
  • Add: unsafe_for_n APIs (cb334c4)
  • Add: NUMA-aware N-Body benchmarks (51c1e9e)
  • Add: QoS levels and allocation_result (7eaab59)
  • Add: Draft new NUMA design (ec090cb)
  • Add: Draft C API (2906295)
  • Add: Draft numa_thread_pool (150220a)
  • Add: Platform-specific yields (81f0812)
  • Add: NUMA allocator (291f3eb)
  • Add: Sleeping mood (8759a08)
  • Add: C++ 20 concepts (e6d62aa)
  • Add: Draft exclusive spawning (a1eb5b9)

Patch

  • Improve: Allow naming pools in C (751cbe5)
  • Docs: Add Rust docs (0386b7d)
  • Fix: Rust bindings (b3464c5)
  • Fix: Non-NUMA C builds metadata APIs (339464a)
  • Improve: Check every HP size for NUMA allocs (275b18d)
  • Fix: Compilation issues (4fd324d)
  • Improve: Redo the Rust lib as a binding (dd4671f)
  • Docs: Naming HW-specific "yielders" (60bce68)
  • Improve: Avoid HugeTLBfs (c496dfe)
  • Fix: colocated_thread_t explicit construction (1b8568c)
  • Docs: Document the C 99 API (8655753)
  • Improve: Propagate colocated_thread (be060aa)
  • Fix: Missing <cstdio> include (eca3f5d)
  • Make: Pre-compile the C library (76c22a1)
  • Improve: New "arch" detection macros (4abd1b6)
  • Fix: Working TPAUSE implementation (662b74d)
  • Make: Disable TSan to avoid stalls on NUMA machines (c5f49e5)
  • Fix: Missing shift to thread-local range (e1d7eac)
  • Fix: Using sized CPU_SET operations (00b56c8)
  • Fix Resetting counters between distributed dynamic runs (432608a)
  • Fix: Compiling distributed pools (5d8a597)
  • Make: Simpler HugeTLB-FS lookups (25874c9)
  • Improve: Separate UMA colocation & NUMA distributed tests (2000849)
  • Fix: Thread ID offset for colocated_pool (240a3f2)
  • Fix: Require passing callbacks by reference (cae247a)
  • Docs: New distributed pool API draft (13d7d2a)
  • Docs: HugeTLB & NUMA deps (54542fa)
  • Fix: Passing compilation & non-NUMA tests (d639085)
  • Improve: Move unsafe_broadcast call into "joiner" (881eb54)
  • Improve: Locate "joiner" lifetime issue (e47d37c)
  • Make: Include missing HugeTLB deps (00c4a5c)
  • Fix: Switch from fu:: global funcs to members (0e9036f)
  • Improve: Print RAM page settings (0ae1bcc)
  • Improve: Better requires constraints (783894c)
  • Improve: Enforce atomics align in for_n_dynamic (2a19ea8)
  • Improve: Move main-thread loop into join (e79579b)
  • Make: Drop C++11 compatibility (7f1a52a)
  • Improve: Return for_n joins (e4d8901)
  • Improve: Minimalistic C interface (c05dc88)
  • Docs: Draft NUMA design (e7646ad)
  • Improve: Addressing global symbols (6e259e5)
  • Fix: Reset main thread affinity (91e1b3b)
  • Improve: Instantiate the templates (0c0ab7a)
  • Make: Disable 128-byte arg ABI warning (d1539a3)
  • Improve: Extend tests to NUMA pools (55003ba)
  • Fix: Partial initialization failure paths (0df0011)
  • Fix: Loading with release semantics (0313ee7)
  • Fix: Extend lifetime of joins (996925c)
  • Fix: Sleeping logic in the default pool (0602965)
  • Fix: Compilation errors (90c8df1)
  • Fix: Passing thread ID to workers (d98ca88)
  • Improve: Freeze 128-byte alignment (8ad5f70)
  • Make: Link & set FU_ENABLE_NUMA from CMake (b6dc5a3)
  • ImprovE: Log NUMA topology in tests (6444972)
Assets 2
Loading