Performance on NUMA systems

[jodavies]

Following discussions with @magv and @nikolai-husung at the 2026 workshop, we can collect some data here regarding the performance impact of running FORM on NUMA systems in different configurations.

To start, here are numbers from my 7900X system, which is not NUMA w.r.t DRAM access, however it is structured as 2x6-core CCDs, each of which has 32MB of L3 cache. Cross-CCD cache access is "slow" (comparatively). I run the "default" configuration (tform-master) and also run with threads confined to a single CCD (numactl --physcpubind=0-5 tform-master) and threads split between CCDs (numactl --physcpubind=0-2,6-8 tform-master). During the sortbot phase of tform, more CPUs are briefly used than the requested numbers of workers; hence the "default" unconfined runs are universally faster than the confined runs. In the confined runs the individual tform threads are allowed to migrate between the allowed set of cores.

Here we already see some large performance differences for some benchmarks. Keeping all threads on a single CCD (which means the total available L3 is halved, but there is no cross-CCD communication) is better than splitting them. Presumably, on systems which are genuinely NUMA w.r.t. DRAM, differences will be even more pronounced.

Timings

Speedup w.r.t tform-master -w6

Benchmark	numa60-tform-master -w6	numa33-tform-master -w6
chromatic	0.97 ± 0.00	0.94 ± 0.01
color	0.97 ± 0.01	0.90 ± 0.01
fmft	0.98 ± 0.01	0.94 ± 0.01
forcer	0.95 ± 0.01	0.83 ± 0.01
forcer-exp	0.94 ± 0.01	0.82 ± 0.01
hyperform	0.95 ± 0.01	0.87 ± 0.01
ibp-box2l	0.98 ± 0.01	0.92 ± 0.02
ibp-mbox1l	1.00 ± 0.03	0.97 ± 0.04
ibp-npbox2l	0.99 ± 0.01	0.96 ± 0.01
ibp-nptri3l	0.99 ± 0.00	0.97 ± 0.01
ibp-pent1l	0.99 ± 0.04	0.97 ± 0.05
ibp-tri3l	0.99 ± 0.01	0.97 ± 0.01
minceex	0.98 ± 0.02	0.94 ± 0.01
mincer	0.98 ± 0.01	0.96 ± 0.01
mzv-dm	0.95 ± 0.01	0.72 ± 0.01
sort-disk	1.00 ± 0.01	1.00 ± 0.01
sort-large	1.01 ± 0.02	0.97 ± 0.08
sort-small	1.00 ± 0.02	0.98 ± 0.02
trace	1.00 ± 0.01	1.02 ± 0.01

The following is as above, but also allowing access to the corresponding SMP threads, i.e. by running with numactl --physcpubind=0-5,12-17 and numactl --physcpubind=0-2,6-8,12-14,18-20. This closes the gap to a large extent, but some benchmarks still have a very obvious impact.

Timings

Speedup w.r.t tform-master -w6

Benchmark	numa60-tform-master -w6	numa33-tform-master -w6
chromatic	1.00 ± 0.01	0.99 ± 0.01
color	1.01 ± 0.01	0.97 ± 0.01
fmft	1.00 ± 0.01	0.98 ± 0.01
forcer	1.00 ± 0.00	0.95 ± 0.00
forcer-exp	1.00 ± 0.00	0.94 ± 0.00
hyperform	1.00 ± 0.01	0.97 ± 0.01
ibp-box2l	1.00 ± 0.01	0.98 ± 0.01
ibp-mbox1l	1.00 ± 0.03	0.99 ± 0.03
ibp-npbox2l	1.00 ± 0.01	0.99 ± 0.01
ibp-nptri3l	1.00 ± 0.01	0.99 ± 0.01
ibp-pent1l	0.99 ± 0.04	0.98 ± 0.04
ibp-tri3l	1.00 ± 0.00	0.99 ± 0.00
minceex	1.00 ± 0.00	0.98 ± 0.01
mincer	1.00 ± 0.00	0.98 ± 0.01
mzv-dm	1.02 ± 0.01	0.85 ± 0.01
sort-disk	1.00 ± 0.01	1.00 ± 0.01
sort-large	0.99 ± 0.02	0.94 ± 0.07
sort-small	1.01 ± 0.02	0.99 ± 0.02
trace	1.00 ± 0.01	1.02 ± 0.01

[nikolai-husung]

Whenever one runs via a queueing system like Slurm, Condor, etc. the scenario of having more cores than workers would need to be requested explicitly and seems unusual to me. On top of that, one may run alongside competing jobs potentially saturating the inter-socket/-CCD bandwidth. The following numbers ignore the latter detail as I ran on a full node with core-usage restricted by taskset:

CPU: 2x AMD EPYC 7302 16-Core Processor (total of 32 physical cores, no hyperthreading)
Memory: 512GB DDR4 3200Mhz

numactl --show
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
cpubind: 0 1
nodebind: 0 1
membind: 0 1
preferred:

numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 257330 MB
node 0 free: 248182 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 1 size: 257992 MB
node 1 free: 250982 MB
node distances:
node 0 1
0: 10 32
1: 32 10

Benchmark	taskset 0xf000f tform -w8	taskset 0x1fc001 tform -w8
chromatic	1.05 ± 0.00	1.07 ± 0.01
color	1.03 ± 0.01	1.04 ± 0.01
fmft	1.04 ± 0.01	1.06 ± 0.00
forcer	1.07 ± 0.00	1.11 ± 0.01
forcer-exp	1.06 ± 0.01	1.09 ± 0.01
hyperform	1.09 ± 0.01	1.12 ± 0.02
ibp-box2l	1.02 ± 0.01	1.03 ± 0.01
ibp-mbox1l	1.02 ± 0.04	1.01 ± 0.02
ibp-npbox2l	1.02 ± 0.01	1.01 ± 0.01
ibp-nptri3l	1.01 ± 0.00	1.02 ± 0.00
ibp-pent1l	1.02 ± 0.05	1.02 ± 0.04
ibp-tri3l	1.01 ± 0.00	1.02 ± 0.00
minceex	1.08 ± 0.04	1.07 ± 0.00
mincer	1.03 ± 0.01	1.05 ± 0.01
mzv-dm	1.16 ± 0.02	1.19 ± 0.07
sort-disk	1.01 ± 0.00	1.00 ± 0.00
sort-large	1.01 ± 0.01	1.02 ± 0.01
sort-small	1.01 ± 0.10	1.01 ± 0.02
trace	1.01 ± 0.01	1.01 ± 0.01

Here, the reference is taskset 0xff tform -w8, i.e., choosing all CPUs on the same socket while 0xf000f has 4 CPUs on each socket and 0x1fc001 has 7 and 1 CPU on the two sockets. 0x1fc001 is the worst case scenario. I highlighted the few cases where the placement makes a significant difference in bold.