Fix: Atomics and Locks (ARM, AArch64, X86)

[ysbaddaden]

This patch should fix the use of atomics in locks once and for all (until we add new architectures, that each will need to be investigated). All usages should now be correct as per the textbook, instead of relying on hacks without much impact.

Explanation

CPU architectures with a weak memory model (such as ARM) can reorder the CPU instructions at runtime (unlike X86), not following the exact order of the assembly. We must respect the memory model guarantees to ensure that we have proper barrier or specific memory ordering at specific places when they are required (e.g. locks).

A lock must ensure that anything after the lock was acquired aren't executed before the lock is acquired, and the other way around: anything before we release the lock must be executed before we actually release it. Each architecture has different quirks:

• AArch64: atomics guarantee the ordering but we must use explicit atomics everywhere (at least load/store with acquire/release ordering) (source);
• ARM v6/v7: atomics don't guarantee any ordering (only the atomicity of the operation), there must add explicit memory barriers around locks (help wanted: can someone test on ARMv6 or v7 hardware? 🙇) (source);
• X86: it seems we can go away with a non atomic store, yet using acquire/release ordering isn't any slower than a non atomic store.

Explicit orderings

X86 is the main reason I added specific orderings. I couldn't notice differences in AArch64 between seq-cst and acq/rel, but on X86 it had a great impact, being no slower than the lazy store while using correct atomics.

Note: I'm following the algorithms used by the Linux v4.4 kernel overall.

Performance impact

I ran the "mutex" example from #14272 by @jgaskins ten times at different MT levels (ranging from 1 to 128 threads) and charted the average value. See run.cr.

On X86 (intel haswell i7-4712HQ, 4 cores, 8 threads) we see that the current patch isn't any slower than the current master (incorrect but appears to work), whereas just using load + store with the default sequentially consistent ordering has a huge impact.

On AArch64 (Neoverse N1 server, 160 cores) we can see that this patch is slower than master with the traditional LL/SC atomics (inherited from ARMv6), but that the difference becomes unnoticeable when the new LSE atomics are enabled (--mattr=+lse) which significantly outperform LL/SC on this hardware.

Future evolution: we might want to enable LSE atomics by default for AArch64 when available: can detect lse feature with LLVM 15 and this snippet by @HertzDevil and maybe a flag to disable when needed.

Help Wanted: I'd be very interested to see results of LL/SC vs LSE atomics for other AArch64 CPUs: Apple, Raspberry Pi, AWS, ... 🙇

Related issues:

• Bug in atomic operations on aarch64 with multi-threading #13010
• Add memory barrier to Mutex#unlock on aarch64 #14272

[Sija]

macOS CI is still failing with Assertion failed: (isValidFailureOrdering(Ordering) && "invalid CmpXchg failure ordering"), function setFailureOrdering, file Instructions.h, line 612.

[ysbaddaden]

Thanks Sija, I missed that assertion, it felt like nixpkgs was crashing. I have no idea what's happening there. The assertion is pointless and doesn't hint at any issue.

Not having the assertion and having the actual LLVM IR error would actually be much more helpful.

Might Crystal::LLVM_VERSION be invalid or something?

[ysbaddaden]

@Sija are you sure? CI reports a different message:

invalid CmpXchg failure ordering

Not

must be atomic

[ysbaddaden]

I tried to reproduce with Nix on Linux but the assertion won't reproduce 🤷

[Sija]

It's odd, failing assertion seems to be this one: https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/IR/Instructions.h#L623-L624, yet the CI message mentions line 612...

Also, llvm-config is at version 11.1.0 (Using /nix/store/pj58mky3176z3xh7xgsp5sf46dfpyy5k-llvm-11.1.0-dev/bin/llvm-config [version= 11.1.0]), but for this version assertion line is different: https://github.com/llvm/llvm-project/blob/llvmorg-11.1.0/llvm/include/llvm/IR/Instructions.h#L611-L612

[ysbaddaden]

The isValidFailureOrdering function doesn't even exist for LLVM 11. It's like the reported and used LLVM versions are completely wrong. Is nix doing shenanigans? 🤔

I think I'm starting to understand: LLVM with assertions turned on still checks that the failure ordering can't have release semantics, but without assertions it will accept them. Sigh, so which is which? 😕

[ysbaddaden]

As stated in the latest commit: I removed the check for the LLVM version and only enable the strictest of choices. That seems to have fixed the issue with LLVM assertions, and I think it's better to not have different semantics based on the LLVM version Crystal happens to have been compiled with.

[HertzDevil]

The assertion failure comes from LLVM 15.0.2, since that is the version the compiler itself is built with.

[ysbaddaden]

Reading again https://llvm.org/docs/Atomics.html I realize the notes for code generation mentions using explicit fences on weak architectures... and I'm back to wonder if we ever needed to put explicit Atomic.fence for ARM at all, because the codegen should already do so.

If so, the problem was merely the lazy set that was breaking the acquire/release requirement (an acquire is only guaranteed when paired with a release, and vice versa).

Let's cross compile & disassemble to check that.

[ysbaddaden]

I got mixed results. With a mere --target=arm-linux-gnueabihf LLVM assumes I'm aiming for an old ARM CPU, and expects __sync_* symbols from libgcc and libgcc_s, and I'm not sure about the memory barriers (depends on the __sync lib).

If I enable a recent CPU (e.g. --mattr=+armv7-a,) then it generates proper load/store with memory barriers (dmb ish). For example:

   304bc:       e3a00f72        mov     r0, #456        ; 0x1c8
   304c0:       e58d0000        str     r0, [sp]
   304c4:       f57ff05b        dmb     ish
   304c8:       e59d0000        ldr     r0, [sp]
   304cc:       f57ff05b        dmb     ish

[ysbaddaden]

Confirmed: the explicit fences are not needed for arvm6 and armv7, while armv5 and older don't support fences. See #14567