[FEATURE]: allow failure for workflow tasks

[salotz]

Feature Request

Background / Motivation

In some cases I want to run a workflow task that I know might fail for various reasons (see #6076 but also others).

What should the user be able to do?

In the workflow step you should be able to specify whether a step is allowed to fail without the entire workflow exiting as failure.

Why do they want to do this? What problem does it solve?

Suggested Implementation(s)

kind: Workflow
steps:
  - name: can-fail
    command: [...]
    allowFailure: true

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[stefreak]

Great feature request 👍 Thank you @salotz

[alex-kattathra-johnson]

Hey there! I would like to give this a shot!

[alex-kattathra-johnson]

@salotz, @stefreak brought up a good point to clarify. Which behaviour do you think is appropriate? Should subsequent steps after the errored step be skipped unless when: always or when: onError are set? The issue description suggests that the current implementation suffices.
#6114 (comment)

[salotz]

I think that an allowFailure: true should basically behave as if it succeeded, meaning subsequent steps should proceed as normal. An allowed failure should not stop running onSuccess tasks and onError should not run.

I think the more ideal solution is to distinguish onError and onFailure. onError would trigger when an allowFailure: true task errors is allowed to pass and equally if it isn't allowed to pass. Whereas a onFailure would only trigger if an allowFailure: false step has an error.

But this probably digresses into more systemic design issues of the Workflow type which is beyond the scope here.

[stefreak]

I like the idea of distinguishing between onError and onFailure. Another problem that complicates things further is that we should also distinguish between "completed with error, but allowed to fail" and "failed" in the Cloud UI (https://app.garden.io) - but I think it is acceptable in the first iteration of this to solve the problem by making it clear in the logs.

I would be open for implementation of this functionality, marking the allowFailure flags and the onFailure behaviour as experimental for a while so we can monitor the usefulness in practice and can make a breaking change if needed without bumping Garden to 0.14. (Similar how we did it with Cloud Builder by adding something like **Stability: Experimental**. Subject to breaking changes within minor releases. to the docs.

How do you feel about this? @salotz @alex-kattathra-johnson

[salotz]

That is fine with me. Like I said I'm fine without those extra features and can't say I would even use them immediately, but seems to be the next logical step. If there is going to be some next major version of Workflow features I'd probably have a bunch more things to consider the issue holistically.