Ensure paused tasks don't run

[jlowin]

Currently, tasks that start off in a PAUSED state will be [correctly] treated as PENDING tasks and run.

This PR changes that so PAUSES do not pass the is_pending() check.

In order to run a PAUSE task, you must put it in a different state (like RESUME)

(depends on #541)

[cicdw]

I'm on board with the end result of this PR, but I'm a little concerned that the is_pending pipeline method does something different than the is_pending State method --> should Paused be a subclass of Running?

[jlowin]

You know, that’s an excellent suggestion. It would automatically work. Is it weird for Paused to be running, but Resume to be pending?

[cicdw]

Is it weird for Paused to be running, but Resume to be pending?

It's quirky, but I think that having the states respond to the pipeline correctly without any special-casing is preferable, because that will ultimately make it easier to reason about stateful behavior. Definitely open to push back though if there are other edge cases I'm not considering!

[jlowin]

So I do have a possible reason for Paused to remain a Pending subclass, which is that only Pending states have cached_inputs.

However, the task isn't actually run until it's put into a Resume state, so the Paused state's inputs don't actually matter -- in fact they need to be transferred to the Resume.

I think this ties in to our ongoing discussion about resulthandlers.

[cicdw]

Hmm yea, good point! However, Paused states can also be created by a user raising a PAUSE signal inside a task, i.e., after a run has started. In this case we would need cached inputs.

[cicdw]

Because of that, actually, I’m OK with leaving Paused states as subclasses of Pending and have a special case for it 👍🏻

[jlowin]

Now rebased on top of #541

[cicdw]

great