[client-shell] Require --force to rerun successful or scheduled tasks

[djmitche]

From #188

@escapewindow
For rerun, we specifically want to check for status in ['failed', 'exception'] and require --force to rerun completed tasks.

---

@srfraser
I wonder whether there is also value in checking a task's dependencies for some operations. For example, if Task C depends on tasks A and B, and we want to rerun C, that doesn't really make sense if A and B didn't complete.

[djmitche]

Simon's comment would be about a task in the "scheduled" state. I'm not even sure the queue will let you rerun such a task (since it hasn't run yet).

[escapewindow]

I know releaseduty has used the fact that cancel/rerun doesn't check for dependency statuses as a way to trigger tasks that have failed or unfinished dependencies. I see this as a bug, but since there's no other way that I know of to do this, it's a "feature".

I think checking for the deps is the proper thing to do. I think allowing for the other behavior may help releaseduty until we stabilize things to the point that we never have tasks that need to trigger before dependencies complete successfully.

[djmitche]

Hm, probably scheduleTask is the right API method there, and it's a queue bug that rerunTask will run a scheduled task. @jonasfj what do you think?

[escapewindow]

You have to cancel it first before rerun will work. And then once you rerun the cancelled task, it runs without deps finishing. As mentioned above, we rely on this behavior, but it's probably not as designed.

[djmitche]

Ah, that makes more sense -- so this is a rerun from the "exception" status.

[escapewindow]

A sheriff just reran successful release tasks again, because treeherder showed them as busted (since treeherder doesn't hide old runs on rerun). This is problematic because Chain of Trust reads the latest run's artifacts for cot verification - and a rerun hides the old results.