Alternative to wake_with()

[nyh]

Right now, code like condvar_wake_one/all use wake_with() to allow waking a thread which might exit as soon as we wake it.

Avi doesn't like wake_with() for two reasons: It is unorthodox API (so users need to know to use it, when and how), and it slows down every wake() with two atomic fetch-add.

Why does condvar_wake_one/all() need to use wake_with at all?
It already does
m.lock();
waiter = wr->t;
wr->t = nullptr;
waiter->wake();
m.unlock();

If the waiter would regain the lock when waking up, then we would know the thread cannot exit before wake_one/all releases this lock, and the problem would be solved. But we didn't regain the lock (except in the case of timeout), because we feared that the waiter will often go back to sleep immediately after waking, to wait for the lock which the waker is still holding.

The idea (suggested by Avi) is, instead of finding wake_with() as a replacement to the above solution - just make the above solution efficient.

As explained above, the waiter will regain the lock after waiting. The waker will do:
m.lock();
preempt_disable();
waiter->wr->t;
wr->t = nullptr;
waiter->wake();
m.unlock();
preempt_enable();

If the waiter thread is on the same CPU as the waker, this is efficient - the waiter will not start to run (and try to regain the lock) until the preempt_enable() after we release the lock. This is more efficient than the wake_with() case because now we have a thread-local increment (preempt_disable()) instead of the atomic increment used by wake_with().

If the waiter thread is on a different CPU, the preempt_disable() won't help, and it can start running before we finish m.unlock(). In the usual case, it will take a short time until m.unlock() completes, so all we need to do is to add to mutex_lock() the ability to do a bit of spinning before going to sleep. We want this feature anyway. This will be a performance improvement over wake_with() only if in the typical case, m.unlock() will finish more quickly than the waiter thread starting to run so we'll have no spinning iterations at all. We'll need to verify that this is actually the case - otherwise all of this won't be an improvement at all.

[nyh]

Guy suggested that:

1. He likes the idea of wake() inside the lock.
2. He suggests there's no need to add the preempt_disable/enable() - instead the scheduler should make better decisions - the waker thread should not be preempted just because the woken waiter has a low vruntime (because it slept for a while). E.g., we need a "minimum time slice" (e.g., 1ms) and not preempt the waker before such time has passed, even if higher-priority threads are woken.
3. In the different-CPU case, since we anyway believe/assume that usually no spinning is needed, we don't even need to implement spinning - if the sleep-on-wakeup situation will be very rare, we don't care if it will be slow.

So basically the only change we need to make to code is to improve the scheduler's handling of this case
(and of course, start using the new idiom of wake-inside-lock).

NOTE: In the condvar case, using this new idiom will mean regaining the lock at the end of the wait. Since the mutex code also uses atomic increments (and more) this will offset any performance saving of not using wake_with(). So I don't believe this new idiom will give any better performance. It will be perhaps just easier for users to use.