Threads leaked if executor is not shut down

[rohanpm]

The RetryExecutor and probably some others will leak threads if the executor is not shut down. Although it could be argued that the caller should shut down the executor, the concurrent.futures ThreadPoolExecutor manages to clean up threads without requiring a shut down, and it would be convenient if that worked for all of our executors too.

Demo script here: https://gist.github.com/rohanpm/117dd79f7fa1e806f721e54d66f67773

Produces output:

============== thread pool ======================
Created 10 futures
------------- threads ----------------
  MainThread daemon=False alive=True
  ThreadPoolExecutor-0_0 daemon=True alive=True
  ThreadPoolExecutor-0_2 daemon=True alive=True
  ThreadPoolExecutor-0_1 daemon=True alive=True
  ThreadPoolExecutor-0_3 daemon=True alive=True

About to GC 0
After GC 0
------------- threads ----------------
  MainThread daemon=False alive=True

About to GC 1
After GC 1
------------- threads ----------------
  MainThread daemon=False alive=True

About to GC 2
After GC 2
============== thread pool w/retry w/shutdown ==============
Created 10 futures
------------- threads ----------------
  MainThread daemon=False alive=True

About to GC 0
After GC 0
------------- threads ----------------
  MainThread daemon=False alive=True

About to GC 1
After GC 1
------------- threads ----------------
  MainThread daemon=False alive=True

About to GC 2
After GC 2
============== thread pool w/retry ==============
Created 10 futures
------------- threads ----------------
  MainThread daemon=False alive=True
  RetryExecutor daemon=True alive=True
  ThreadPoolExecutor-2_2 daemon=True alive=True
  ThreadPoolExecutor-2_1 daemon=True alive=True
  ThreadPoolExecutor-2_0 daemon=True alive=True

About to GC 0
After GC 0
------------- threads ----------------
  MainThread daemon=False alive=True
  RetryExecutor daemon=True alive=True
  ThreadPoolExecutor-2_2 daemon=True alive=True
  ThreadPoolExecutor-2_3 daemon=True alive=True
  ThreadPoolExecutor-2_1 daemon=True alive=True
  ThreadPoolExecutor-2_0 daemon=True alive=True

About to GC 1
After GC 1
------------- threads ----------------
  MainThread daemon=False alive=True
  RetryExecutor daemon=True alive=True
  ThreadPoolExecutor-2_2 daemon=True alive=True
  ThreadPoolExecutor-2_3 daemon=True alive=True
  ThreadPoolExecutor-2_1 daemon=True alive=True
  ThreadPoolExecutor-2_0 daemon=True alive=True

About to GC 2
After GC 2

With ThreadPoolExecutor alone, or with ThreadPool & retry & shutdown, threads created by the executors disappear. However, if retry is used and shutdown is not used, threads remain alive forever.

The ThreadPoolExecutor implementation shows careful usage of weakref allowing the executor to be collected while the threads are running. Presumably same approach should be used for all cases where our executors create threads.