3-state sleep

[sberkun]

In order to avoid spinlocks, threads need a mechanism to sleep themselves and wake other threads. This "3-state sleep" allows any thread to be in one of the following states:

• asleep
• awake
• caffeinated: awake, and sleeping this thread will set the state back to awake

The caffeinated state exists so that a thread may sleep, while another thread wakes it, without any fear of race conditions. If the thread sleeps first, then it is awoken by the other thread, then it will sleep, then continue running in the normal/awake state. If the other thread wakes it first, then it sleeps, it will also end up in the normal/awake state.

The biggest unanswered question is how this should interact with other sources of sleeping (specifically, the delay until instruction). The simplest answer is to just say that this may not be used in conjunction with other sleep sources (i.e. if you sleep/wake a thread, that thread may not use delay_until), but there might be a more elegant solution.

TODO:

• add assertions
• write tests
• modify threading runtime to take advantage of 3-state sleep

[sberkun]

@lsk567 @erlingrj this isn't fully fleshed out yet, but I wanted to know your thoughts:

• so far, this allows a thread to sleep when not in use, or waiting on a software lock. Will probably allow sleeping on other software primitives as well (semaphores, condition variables). See flexpret_lock.c for example implementation.
• runtime still relies on spinlock for acquiring the hardware lock. Deadlock may occur if a SRTT holds the hardware lock, then all the HRTTs spinlock waiting for it. Given this, should the hardware lock be changed to an implementation that sleeps?