Explore: Move mold promotion logic in workers

[casperisfine]

Right now, it's the master process that keeps an eye on workers and chose to promote one of them when some condition is reached.

Pros:

• It's somewhat easy to make it race-condition free because only a single threaded process makes the decision.

Cons:

• It prevent the master from sleeping for prolonged amount of time, like in unicorn, as we have to wake up frequently to check if the promotion condition was reached. That's not great.
• The condition has to be checked on all workers sequentially, which means it doesn't scale very well with the number of workers.
• We don't really know if the worker is idle of busy with a request that will still take a very long time. So we may end up selecting a busy worker that will end up timing out. We currently handle this case, but it delays the new generation.
• It makes it harder to allow custom mold selection implementations, as the master doesn't load the app.

Since that's a lot of cons, I think we should explore moving this logic into the workers.

Pros:

• Workers would only be considered for promotion when they are idle (or just finished a request).
• That reduce load on the master.

Cons:

• We need some shared mutex/semaphore to make sure only one worker is promoted, but that should be doable with a Raindrop atomic counter.

The logic would be something like:

def check_promotion
  if condition_met
    if acquire_promotion_lock
      start_promotion # start by notifying master, promote, confirm to master.
    else
      # somehow debounce promotion_check ?
      # e.g. if we continue to use request_count, we can do `request_limit *= 1.20`
    end
  end
end