Pessimisation fix for condition_variable notify_*

[termoose]

This should fix #1284. Some of these locks are for setting stopped and started flags, those functions could get a cleaner look by using std::atomic_bool on those flags and avoiding the explicit lock/unlock.

[cryptocode]

LGTM

[clemahieu]

I was trying to apply lock pessimisation avoidance before but in some case it seemed to create a race condition though how that would happen escapes me now.

As long as all shared variables that are modified are protected by the same mutex it should be safe to notify the condition outside the lock right?

Does this pass with a tsan run?

[cryptocode]

Should be safe and beneficial: "The notifying thread does not need to hold the lock on the same mutex as the one held by the waiting thread(s); in fact doing so is a pessimization, since the notified thread would immediately block again, waiting for the notifying thread to release the lock."

[termoose]

Yes this is the way to go and should increase performance in some cases. It has been discussed here.

Unfortunately I'm not able to run Clang's ThreadSanitizer here on my Mac as it's not supported for some reason.

What do you think about using std::atomic_bools for those flags? Should that be a separate issue? I'd be happy to look into it. Will for example turn code like this from node.cpp:

  std::unique_lock<std::mutex> lock (mutex);
  started = true;

  lock.unlock ();
  condition.notify_all ();
  lock.lock ();

into this

  started.store(true); // can use std::atomic::load to make it explicit that we're dealing with atomics
  condition.notify_all ();
  std::unique_lock<std::mutex> lock (mutex);

[clemahieu]

Since the condition variable is tied to the mutex we can't avoid holding a mutex over setting the variable, for instance

T1 Check started (false)
T2 Set started (true)
T2 Notify listeners
T1 Wait on condition variable (stalled)

[clemahieu]

Looks great, thanks!