Cleanup pending pods on scale down

[BitTheByte]

Currently, the operator retires workers using the HTTP or RPC APIs however those only control the connected dask workers, the operator should take into count dask's Kubernetes worker pods that are in a pending state as those will cause a useless Kubernetes cluster scale-up and then connect to dask and get retired thus a scale down should retire active workers and prevent pending pods from entering running state

[jacobtomlinson]

Agreed.

We could add a check here for any Pods that aren't in a Running phase and delete those before calling retire_workers (if that's even necessary any more).

dask-kubernetes/dask_kubernetes/operator/controller/controller.py

Lines 599 to 600 in 92714da

	if workers_needed < 0:
	worker_ids = await retire_workers(

[BitTheByte]

Looks good to me, we should also subtract pending workers from the number of workers passed to retire_workers