Support non-atomic updates on atomics

[ronawho]

Atomicity is part of the type in Chapel and there's currently no way to do non-atomic operations on atomic variables. However, many HPC applications tend to have phases where only some need atomicity and others only have non-conflicting updates (a simple case is HPCC RA where initialization is race-free, the kernel requires atomics, and verification is read only.) Atomic operations have a non-trivial overhead for processor atomics and especially for network atomics, so being able to perform non-atomic operations in some phases could have some dramatic performance benefits.

I've been imagining this as doing non-atomic updates on atomics, but an alternative would be a mechanism to support atomic updates to regular non-atomics vars. As examples in other languages:

• Rust supports get_mut, which returns a mutable reference to the underlying base type. I think this only works because rust only supports atomics on platforms that have hardware support, so they can always implement atomics with native word sizes.
• C++20 added an atomic_ref, which allows an atomic reference to a non-atomic object (though I think that underlying non-atomic object has alignment requirements, which comes back to CPU requirements for atomic ops)

The main implementation challenge is that there's hardware alignment requirements to support CPU and NIC atomics. Practically speaking I think the alignment requirements are the same between processor and network atomics so at the hardware level doing non-atomic updates on the same memory locations shouldn't be too bad. And even having processor atomics updates to memory that was used for network atomics should be feasible, but none of these operations are coherent, so you have to make sure things are quiesced/fenced/whatever before switching phases" Currently, we represent everything with different types even at the runtime level, so we might need compiler and backend changes to support reusing memory locations for different operations.

The inverse (supporting atomics operations on non-atomic variables) is I think harder because we don't have the same alignment guarantees from non-atomic variables. I think C++ atomic_ref gets around this by using a lock to protect accesses if the alignment isn't there, but that sounds quite expensive (basically need a hashtable of locks to map an address to a lock you can use).

Note that a "workaround" for this is to have an atomic and non-atomic array that you swap between during different phases. This isn't ideal though because it has a high overhead both in terms of memory usage and the cost to copy from one array to the other (and even in this case we'd want a non-atomic way to read from or write to the atomic array.)

[mppf]

The main implementation challenge is that there's hardware alignment requirements to support CPU and NIC atomics.

AFAIK, these aren't really a problem with the "non-atomic update on atomic" approach. The memory will be appropriately aligned to allow atomic access. Even if we implemented the atomic with a lock, there is still a value in there somewhere we can work with (without taking the lock) and potentially even return a reference to.