thread_pool: fix race condition when shutting down workers

[unleashed]

The following backtrace appeared in my error logs:

Exception handling servers: undefined method `join' for nil:NilClass (NoMethodError)
/var/lib/gems/2.2.0/bundler/gems/puma-2.14.0/lib/puma/thread_pool.rb:250:in `shutdown'
/var/lib/gems/2.2.0/bundler/gems/puma-2.14.0/lib/puma/server.rb:814:in `graceful_shutdown'
/var/lib/gems/2.2.0/bundler/gems/puma-2.14.0/lib/puma/server.rb:338:in `handle_servers'
/var/lib/gems/2.2.0/bundler/gems/puma-2.14.0/lib/puma/server.rb:296:in `block in run'

This signals a problem in thread_pool.rb:250, where the code below lives:

if !@workers.empty?
  @workers.first.join until @workers.empty?
end

Since at that point @mutex is not held, @workers can and does in fact get modified by threads shutting down. There is a race condition where the worker disappears between the time of check and the time of use because of the code below in thread_pool.rb:111, when the worker is exiting:

mutex.synchronize do
  @spawned -= 1
  @workers.delete th
end

I have fixed this by duping the worker threads array inside the critical section and using each to iterate and join all workers.