A counter that every thread increments (a request counter, a hit tally, a shared refcount) is the most natural thing to reach for, and one of the most expensive. Each atomic increment has to own the counter's cache line exclusively, so when many threads increment the same counter the line ping-pongs between cores and the increments serialize behind coherence traffic. This measures how bad it is on an Apple M4, and the answer is that it scales negatively: adding a thread makes it slower.
| threads | shared, one counter | private, per-thread counters | shared, cross-cluster |
|---|---|---|---|
| 1 | 542 [534, 543] | 537 [534, 543] | 538 [534, 542] |
| 2 | 260 [252, 260] | 1071 [1061, 1075] | 207 [193, 208] |
| 3 | 84 [78, 85] | 1636 [1602, 1653] | 93 [66, 94] |
| 4 | 74 [54, 95] | 2241 [2197, 2242] | 73 [68, 75] |
| 6 | 56 [52, 61] | 3232 [3169, 3235] | 42 [28, 52] |
| 8 | 37 [35, 37] | 4157 [4139, 4159] | 31 [23, 36] |
Read the shared column top to bottom: two threads (260 Mops/s) are already slower than one (542), and eight threads run at 37 Mops/s, about a fourteenth of a single thread. Adding parallelism subtracts throughput. The private column is the same work sharded onto per-thread counters, each on its own cache line: it scales cleanly, 4157 Mops/s at eight threads, 112 times the shared counter at the same thread count. The cross-cluster column shares one counter across P and E cores, and at higher thread counts it is worse than sharing within a cluster (31 vs 37 at eight), because the contention is resolved across the system-level cache rather than within one cluster's L2.
No increments are lost: relaxed atomics stay correct, they are just slow. The cost is entirely the coherence traffic, not a correctness compromise.
A single hot atomic counter is a scalability trap: it does not just stop speeding up with more threads, it slows down, and it is worse the further apart the threads run. If a counter is contended, do not share it. Give each thread (or each core, or each shard) its own counter on its own cache line and sum them when you need the total. That turns a fourteen-times slowdown into a linear speedup, for the price of a little padding and a final reduction.
./run.sh
Needs clang for arm64 and Python 3 (standard library only; the bootstrap is pure Python). CPU only. Absolute rates vary run to run, but the shared collapse, the private linear scaling, and the cross-cluster penalty do not.
bench/atomcliff.c: the N-thread counter benchmark with QoS placement, a speed probe, and the lost-update oracle.bench/sweep.py: the thread-count sweep for shared, private, and cross-cluster, with BCa CIs.PREREG.md: the pre-registration.run.sh: reproduces everything.
Apple M4, CPU only. The claim is the scaling shapes (shared negative, private linear, cross worse than same) and their magnitudes with CIs, not exact constants.
MIT.