Unblock event loop while waiting for ThreadpoolExecutor to shut down

[fjetter]

This reinstates #5883
which was reverted in #5961 / #5932

I could confirm the flakyness of test_missing_data_errant_worker after this change and am reasonably certain this is caused by #5910 which causes a closing worker to be restarted such that, even after Worker.close is done, the worker still appears to be partially up.

The only reason I can see why this change promotes this behaviour is that if we no longer block the event loop while the threadpool is closing, this opens a much larger window for incoming requests to come in and being processed while close is running.

Closes #6239

[github-actions]

Unit Test Results

       16 files  ±0         16 suites  ±0   7h 25m 8s ⏱️ + 13m 29s
  2 741 tests  - 1    2 656 ✔️  - 4       81 💤 ±0  4 ❌ +3 
21 829 runs  +8  20 777 ✔️ +4  1 048 💤 +2  4 ❌ +2 

For more details on these failures, see this check.

Results for commit 37d9277. ± Comparison against base commit b837003.

♻️ This comment has been updated with latest results.

[mrocklin]

Thoughts on isolating out the unblock commit from the other one? It looks like this PR is based off of #5910 , which seems a bit more complex.

[mrocklin]

I'm fine with this PR. The increased testing failures are concerning. I'm guessing that it's just noise, but we should both take a look at them so that we feel the pain of them in our brains :)

[mrocklin]

This one seems maybe interesting: #6247

[fjetter]

As I described in the above comment, with this change, #5910 becomes more important. However, it seems to come together finally (apart from merge conflicts)