Invalid memory ordering in thread park / unpark

[carllerche]

I'm pretty sure that the memory orderings uses in the thread park / unpark are incorrect. They happen to work on x86 due to the generated ASM being stronger than the spec, but it will doubtfully work on other architectures.

There are a few locations where a store is used with SeqCst assuming that this would "acquire", but this is not the case. For example here

I discovered this issue when a PR was provided to copy the Rust code into Tokio.

See the discussion here

[RalfJung]

I think that particular store is fine. it doesn't need to "acquire".

What really matters is that we have read a NOTIFY with "acquire" semantics, and that happened in the preceeding compare_exchange.

[carllerche]

it doesn't need to "acquire".

Could you explain why you think this is true? The park / unpark API specifically needs to provide acq / rel semantics to the user of the API. When a call to park returns, it must have acquired all writes that happened before calls to unpark that resulted in the unpark.

Specifically, that line in park must synchronize with this line in unpark:

rust/src/libstd/thread/mod.rs

Line 1057 in 6e5e63a

match self.inner.state.compare_exchange(EMPTY, NOTIFIED, SeqCst, SeqCst) {

That would

[RalfJung]

So the point is specifically the case where this CAS fails because the value is already NOTIFIED?

            match self.inner.state.compare_exchange(EMPTY, NOTIFIED, SeqCst, SeqCst) {
                Ok(_) => return, // no one was waiting
                Err(NOTIFIED) => return, // already unparked <--
                Err(PARKED) => {} // gotta go wake someone up
                _ => panic!("inconsistent state in unpark"),
            }

That is interesting indeed. However, notice that if that CAS failed, it performed just a read. So adding an extra read in park will not change anything; only a read reading from a write would make a difference.

However, it is possible that here SeqCst actually makes the difference. I will have to read up on how exactly it affects happens-before.

[RalfJung]

Looking at the formulation in http://plv.mpi-sws.org/scfix/paper.pdf, SC accesses have some extra requirements for their ordering relative to each other but do not seem give extra synchronization otherwise.

However, I am no longer sure what you think the problem is. The line you pointed at, if it actually performs a write, will write the NOTIFIED which is later read by the thread.inner.state.compare_exchange(EMPTY, PARKED) in park. So there is a synchronization happening between the thread causing the unpark, and the waking-up thread.

---

However, the line I pointed at above is somewhat interesting -- and your comments in the other discussion sounds like that is what you are worried about:

No, because a third thread can come along and unpark while the park fn is between 257 and 262. The bug is specifically that the park thread must acquire the memory from a third thread that calls unpark before the park thread hits line 262.

The only situation I can imagine where there is a difference is something like the following:

• Thread 0 parks itself.
• Thread 1 calls unpark. The call completes, state is NOTIFIED.
• Thread 2 calls unpark -- this is a NOP because the state is already NOTIFIED. The call completes.
• Thread 0 notices this, wakes up, proceeds to it's CAS and synchronizes with thread 1. State is EMPTY.

Now thread 0 never synchronized with thread 2. However, notice that thread 0 is also never waiting for thread 2 -- so, it is equally likely that the execution proceeds as follows:

• Thread 0 parks itself.
• Thread 1 calls unpark. The call completes.
• Thread 2 calls unpark, but gets descheduled immediately.
• Thread 0 notices this, wakes up, proceeds to it's CAS and synchronizes with thread 1.
• Now thread 2 continues and completes its unpark, which sets the state back to NOTIFIED and then returns.

So, there is some kind of race here when a thread wakes up while a redundant unpark happens -- depending on the outcome of the race, the thread has a token remaining (state NOTIFIED) or not (state EMPTY). Is that the desired behavior?
(There is yet another trace where first thread 0 does the CAS to EMPTY, then thread 2 does the CAS to NOTIFIED and finally thread 0 writes EMPTY again, eating a token. But this doesn't add any more possible behaviors, I think.)

But anyway no amount of additional ordering can fix this, this is a more fundamental point in the algorithm: How to handle redundant unparks? If these are allowed, we inherently have a race because whether an unpark is redundant or not is racy. And as long as a redundantly unparking thread does not perform any writes (or barriers), there is no way it could Release any of its accesses to the formerly parked thread.

[parched]

Correct me if I'm wrong, but if there a 2 threads doing the unparking, then there's no way the parked thread can know which thread unparked it. So, regardless if every call to unpark synchronized with parked thread, there still needs to be another form of synchronization to tell which thread which thread called unpark. That is, the synchonization here is only needed when there is one unparker, which it currently works fine for?

[comex]

There's still a potential race. If Thread 2 reads NOTIFIED, it knows that Thread 0 will unpark "in the future"; however, without sequential consistency, that doesn't mean Thread 2's prior writes will necessarily be visible to Thread 0 when it does so. If Thread 0 then goes to sleep again, it may never see Thread 2's writes unless someone else happens to unpark it.

However, as far as I know, using SeqCst here is incorrect, as it doesn't guarantee any ordering with respect to other accesses that are not SeqCst.

It does work if the other accesses happen to be SeqCst: that is, if Thread 2 does a SeqCst write to some other variable before calling unpark, and Thread 0 does a corresponding SeqCst read after returning from park, then the use of SeqCst for the compare-exchange should guarantee that Thread 0 reads what Thread 2 wrote. This is something that wouldn't work with just AcqRel.

But the desired semantics for park/unpark are stronger. When Thread 2 calls unpark, it's supposed to guarantee that Thread 0 will 'eventually' return from park and at that point be able to see Thread 2's prior writes, even with a relaxed or acquire load. That guarantee is upheld in the 'successful CAS from EMPTY to NOTIFIED' case, but not the 'unsuccessful CAS because it was NOTIFIED already' case.

[parched]

But the desired semantics for park/unpark are stronger. When Thread 2 calls unpark, it's supposed to guarantee that Thread 0 will 'eventually' return from park and at that point be able to see Thread 2's prior writes.

Ok, I see the desire but I'm having trouble understanding how that could be useful. Do you have a concrete example where it's needed?

[comex]

For example, Thread 2 pushes something onto a lock-free queue, then unparks Thread 0; Thread 0 is running an infinite loop that alternates between calling park() and checking the queue, removing any pending items.

If the memory model is weak enough, I think it should be needed for almost any use of park/unpark. Even when a mutex is involved, it seems that mutexes don't necessarily guarantee sequential consistency (!) … So, for example, even if Thread 2 locks the mutex to write some data, and Thread 0 locks the mutex before reading it, in theory Thread 0 could acquire the mutex before Thread 2 and not see the data, even if Thread 2, after unlocking the mutex, then observes Thread 0 being in NOTIFIED state and thus supposedly not having locked it yet!

[parched]

But the queue would have to be designed so that writer synchronizes with the reader, otherwise you would have a race when Thread 0 was unparked by Thread 1 and then starts reading the queue while Thread 2 is writing to it?

[RalfJung]

But the desired semantics for park/unpark are stronger. When Thread 2 calls unpark, it's supposed to guarantee that Thread 0 will 'eventually' return from park and at that point be able to see Thread 2's prior writes, even with a relaxed or acquire load. That guarantee is upheld in the 'successful CAS from EMPTY to NOTIFIED' case, but not the 'unsuccessful CAS because it was NOTIFIED already' case.

I agree on the analysis of the problem. Not sure about the desired API. ;)

However, this can only be a problem in cases where a thread calls unpark without knowing that the other thread is even parked. That seems like a mis-use of the parking API to me? Isn't there generally one dedicated thread responsible for waking up the parked threads, e.g. the thread that released the Mutex, or the thread that triggered the barrier, or the thread that completed initializing a sync::Once?

[carllerche]

Of course you still need the lock free queue to be synchronized. The issue is that the park thread needs to unpark after unpark is called, or it could unpark before it sees the work done on the lock free queue, thus missing the wakeup.

[RalfJung]

@carllerche So what is an example of a data structure that speculatively calls unpark on a thread that may or may not be parked right now, and relies on synchronization? I believe that is the only case where the issue we are discussing here actually arises.

[carllerche]

I don't understand your question. Are you asking if, given two threads A, B, where A may be blocked in park at some point, does it make sense for B to call unpark without knowing if A is currently blocked?

[RalfJung]

Essentially, yes. And also I am asking if you agree with my analysis above that this is the only case where the issue we are discussing here surfaces.