Improve contended Win64 small-block allocation with opt-in arena affinity

[janrysavy]

Summary

FastMM5 currently starts every small-block arena search at arena 0. Under heavy allocator contention, worker threads repeatedly converge on the same first locks even when other arenas are available.

This change adds an opt-in Win64 path that gives a thread a stable starting arena for a block type after that thread has found every arena locked for that type. It improves the multithreaded speed score on both tested CPUs, while ordinary single-thread allocation becomes slightly slower. The feature therefore remains opt-in through FastMM_EnableAdaptiveSmallBlockArenaAffinity.

The candidate includes PR #98 plus affinity because that is the intended branch stack. The focused Raw Performance tests contain allocation and free without ReallocMem, so those checks exercise affinity rather than PR #98's reallocation fast path.

Implementation

After severe contention, FastMM5 records a compact live-thread key, assigns an arena in round-robin order, and enables affinity only for the contended block type. Later allocations of that type start their first arena scan at the assigned arena and wrap once through all arenas. The second attempt still starts at arena 0, preserving the existing placement of new sequential-feed spans.

Affinity-table replacement is protected by per-slot odd/even versions, so colliding threads cannot publish one thread's key with another thread's type mask. Readers use affinity only when the version is unchanged and even. Missing, stale, or changing entries fall back to arena 0. A per-block-type activation bitmap keeps unrelated allocation sizes on the original fast path.

The implementation is limited to the Win64 assembler path. Win32 and PurePascal builds ignore the option.

Benchmark and results

The benchmark is FastCodeBenchmark 3.0.2 at commit 990f66b4c74e461c0fecc567cf407f885b2604f5, built for Win64 with Delphi 13.1 compiler version 37.0 and release optimization. It uses the normal latest Win64 selection and unchanged benchmark bodies, repeat counts, thread counts, and score weights. The FastMM4-AVX documentation refers to tests from the same FastCodeBenchmark lineage.

The current benchmark no longer emits its historical graph score, so the companion Score-Results.py script applies that scoring model to the retained result rows: reciprocal elapsed ticks, normalization to the best of all five managers per benchmark, aggregation with the registered global weights, and group normalization. The result is then indexed to the Delphi RTL allocator at 100.00% for each CPU and group; higher is better. The table reports the elapsed-time speed component only. Peak memory and each benchmark's speed-versus-memory mix are evaluated separately and do not affect these columns. For example, 134.69% means a speed score 34.69% above RTL, not that every allocation is 134.69% faster.

Methodology - complete current Fastcode Win64 default selection

The benchmark uses every test selected by the current Win64 startup rules: RunByDefault is true and Is32BitSpecial is false. The resulting 53-name selection includes individual allocation and reallocation operations, raw throughput, NexusDB, string and array workloads, and application-derived replays such as Webbroker and eLink. The single-thread column combines Fastcode's single-thread reallocation, allocation/free, and replay categories. Those benchmarks run before the threaded part of the suite, so the column describes pre-contention single-thread behavior; a separate same-process test below covers single-thread work after affinity has activated. Benchmark bodies and registered global weights are unchanged.

Fastcode calculates global weights for all RunByDefault benchmarks before Win64 startup unchecks Is32BitSpecial tests. Preserving those current rules makes multithread replays 68.10% of the selected multithread score instead of the category design's nominal 53.70%. This is a reproducible result for the current Win64 default selection rather than a platform-neutral overall ranking.

CPU	Memory manager	Single-thread RTL-relative speed score	Multithread RTL-relative speed score
AMD Ryzen 9 7950X	Delphi 13.1 RTL	100.00%	100.00%
AMD Ryzen 9 7950X	FastMM4 4.993	102.91%	106.44%
AMD Ryzen 9 7950X	FastMM4-AVX 1.0.12	97.84%	48.46%
AMD Ryzen 9 7950X	Upstream FastMM5	114.12%	87.25%
AMD Ryzen 9 7950X	FastMM5 + PR #98 + affinity	112.32%	134.69%
Intel Core i7-8750H	Delphi 13.1 RTL	100.00%	100.00%
Intel Core i7-8750H	FastMM4 4.993	87.56%	101.13%
Intel Core i7-8750H	FastMM4-AVX 1.0.12	85.82%	74.79%
Intel Core i7-8750H	Upstream FastMM5	97.47%	127.72%
Intel Core i7-8750H	FastMM5 + PR #98 + affinity	97.20%	137.51%

The FastMM5 values use the median of three complete runs per benchmark, while Delphi RTL, FastMM4, and FastMM4-AVX each have one complete retained run per CPU and are orientation baselines only.

The primary patch comparison is the paired three-run FastMM5 result. Complete-suite multithread speed-score changes were +54.90%, +9.76%, and +14.84% on AMD, with a median of +14.84%, and +8.10%, +7.97%, and +7.72% on Intel, with a median of +7.97%. The improvement direction is consistent on both CPUs, while the AMD magnitude is workload- and run-sensitive. The per-benchmark-median table remains useful context but its larger AMD difference should not be read as the typical complete-run improvement.

Complete-suite single-thread speed-score changes were -1.10%, -1.25%, and -4.65% on AMD, with a median of -1.25%, and -1.00%, +0.41%, and -2.34% on Intel, with a median of -1.00%. A focused fresh Raw 1 check was effectively neutral: 0.72% faster on AMD and 0.65% slower on Intel by median elapsed time. After Raw 16 activated affinity in the same process, Raw 1 was 4.79% slower on AMD and 2.74% slower on Intel.

Focused full-duration checks supported the multithread conclusion. All AMD Raw 8/16/31/63 pairs were faster with affinity. On Intel, Raw 16/31/63 was faster in every pair, while Raw 8 was neutral. A separate 52-name historical-selector pair showed the same general tradeoff: higher multithreaded speed with a workload-dependent memory cost.

Alternatives evaluated

Before selecting this implementation, I tested the following designs:

• Stable affinity was simple and fast, but distributed partly used spans across arenas and increased retained memory.
• Returning new-span allocation to arena 0 improved memory convergence but still applied affinity too broadly.
• Contention-triggered activation avoided uncontended work, but deriving the arena from thread-ID bits could cluster workers.
• Round-robin assignment distributed workers evenly, but the first affinity table could mistake a reused thread ID for a live entry.
• A stronger live-thread key fixed ID reuse, but thread-wide affinity still affected unrelated block types.
• Per-block-type activation fixed that scope problem and became part of the selected design.
• A separate wrapper could scan all arenas twice and caused a large multithreaded regression.
• Sharing the wrapper's second attempt removed the obvious double scan, but results depended on earlier benchmark state.
• Colocating the activation flag reduced cache traffic without fixing the wrapper's sequence dependence.
• Dynamic memory-manager entry-point switching proved unsafe.
• A marker in the block-size lookup weakened the useful multithreaded result.
• Separate normal and affinity scanners removed the fresh single-thread loss but still regressed single-thread work after affinity activation.

The selected opt-in design combines contention-triggered activation, round-robin assignment, a live-thread key, collision-safe publication, per-block-type state, and arena 0 convergence for new spans.

Build and validation

The candidate passed the regular Fastcode validation suite and a 24-worker contention-then-validation run. It also passed that sequence with five arenas, compiled with 16 arenas, and passed a one-slot collision stress build followed by Validation 24. Win32 default, Win32 with the option symbol, and Win64 PurePascal compile checks succeeded.

Reproduction

Build upstream FastMM5 at 1a6fb17e4ed525a52c6399ab3e57c7e11ca4d3a3. For the candidate, apply PR #98 rebased as c291b2890075e2d616e86ed7b284b5eba248cf26, then the affinity commits through 49b52fc360238a1ac475b6e4ff7100c47c3dbc37, and define FastMM_EnableAdaptiveSmallBlockArenaAffinity.

Build FastCodeBenchmark commit 990f66b4c74e461c0fecc567cf407f885b2604f5 with Delphi 13.1 Win64 release optimization. Run the normal selection from a directory containing the supplied replay traces. Run each allocator in a fresh process and retain the raw ticks, peak KiB, completion record, and exit code.

Scope

This option is intended for Win64 applications with severe small-block allocator contention. It is not a general replacement for the normal allocation path. Applications that do not accept the small single-thread cost can leave the option undefined.