Remove time.Sleep from run controller delete

[stuartleeks]

After investigating #140 we found a time.Sleep call for 15 seconds in the run controller when deleting a run.

The original flow was to cancel the run, sleep for 15 seconds and then delete the run.

This PR modifies the behaviour: if the run is not in a terminal state then call cancel and requeue for 5s time, if the run is in a terminal state then call delete. For runs that are already terminated this removes the wait completely and proceeds with deletiong. For an active run that takes 6s to cancel the flow will be

• 1st time: get state (running), call cancel and requeue fo 5s time
• 2nd time (after 5s): get state (not yet cancelled), call cancel and requeue fo 5s time
• 3rd time (after 10s): get state (now cancelled) and call delete

With this change in place the load test now runs smoothly for 10 mins (until a different issue occurs that we will address in a future PR). The following graph shows the work queue depth remaining stabe and low: