fix: report failure when at least one of the target is failing

[lemeurherve]

fix: report failure when at least one of the target is failing

Closes #632

Test

To test this pull request, you can run the following commands:

cp <to_package_directory>
go test

Additional Information

Tradeoff

Potential improvement

[dduportal]

LGTM but I defer to @olblak because it might break some use cases that we did not htink about.

[olblak]

LGTM but I defer to @olblak because it might break some use cases that we did not htink about.

If I remember correctly, the motivation to fail early after one failing target was to save some time (api call) but I think your suggestion improves the user experience as we see immediately what target is succeeding and those failing.

[dduportal]

Correct ! Thanks for the memory. If I'm not mistaken, it was beforr you reworked the scm to have pullrequest/scm as top level: the proposal here makes sense now (while it was hard to implement before the rework).

[dduportal]

LGTM (under the assumption that manual testing has to be done)

[olblak]

LGTM (under the assumption that manual testing has to be done)

I'll do some test at the end of the day

[olblak]

There is something fishy/
I tested this PR using

targets:
  1:  
    kind: shell
    spec:
      command: "true"

  2:  
    kind: shell
    spec:
      command: "false"

  3:  
    kind: shell
    spec:
      command: "false"

  4:  
    kind: shell
    spec:
      command: "false"

and I got

REPORTS:


✗ TEST.YAML:
	Sources:
	Target:
		✔ [1]  (kind: shell)
		- [2]  (kind: shell)
		- [3]  (kind: shell)
		- [4]  (kind: shell)

where it stopped after one target failure

[olblak]

@lemeurherve I am opening a pullrequest on this one with a suggestion

[olblak]

Oops sorry I pushed directly to your branch :/, feel free to review my commit

[olblak]

With the manifest that I mention earlier, I get the following output

+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "/tmp/test.yaml"

SCM repository retrieved: 0


++++++++++++
+ PIPELINE +
++++++++++++



#############
# TEST.YAML #
#############


TARGETS
========

4
-

**Dry Run enabled**

The shell 🐚 command "false" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----

3
-

**Dry Run enabled**

The shell 🐚 command "false" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----

2
-

**Dry Run enabled**

The shell 🐚 command "false" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----

1
-

**Dry Run enabled**

The shell 🐚 command "true" ran successfully with the following output:
----
----
No change detected

ERROR: something went wrong in target "4" : "Shell command exited on error."
ERROR: something went wrong in target "3" : "Shell command exited on error."
ERROR: something went wrong in target "2" : "Shell command exited on error."

Pipeline "TEST.YAML" failed
Skipping due to:
	"targets stage:\t\"something went wrong during target execution\""
targets stage:	"something went wrong during target execution"

=============================

REPORTS:


✗ TEST.YAML:
	Sources:
	Target:
		✔ [1]  (kind: shell)
		✗ [2]  (kind: shell)
		✗ [3]  (kind: shell)
		✗ [4]  (kind: shell)



Run Summary
===========
Pipeline(s) run:
  * Changed:	0
  * Failed:	1
  * Skipped:	0
  * Succeeded:	0
  * Total:	1
ERROR: ✗ 1 over 1 pipeline failed
ERROR: command failed

[lemeurherve]

Reviewing your commits, thanks @olblak!

[lemeurherve]

@olblak LGTM 😃

[olblak]

I took the opportunity to fix minor issue

1. Always run all target no matter if an issue occur or not
2. fix small output issue

[olblak]

Please note that we don't handle the case where a target depends on another target which already failed
I am wondering if it's something we want like for source

[olblak]

Please note that we don't handle the case where a target depends on another target which already failed
I am wondering if it's something we want like for source

I am planning to implement the missing depends on ccheck

[olblak]

title: Test Various target scenario

targets:
  1:  
    name: Should be succeeding
    kind: shell
    spec:
      command: "true"

  2:  
    name: Should be failling
    kind: shell
    spec:
      command: "false"

  3:  
    name: Should be failling
    kind: shell
    spec:
      command: "false"

  4:  
    name: Should be skipped
    depends_on:
      - 2 
    kind: shell
    spec:
      command: "false"

  5:  
    name: Should be succeeding
    kind: shell
    depends_on:
      - 1 
    spec:
      command: "true"

output

+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "e2e/updatecli.d/command.yaml"

SCM repository retrieved: 0


++++++++++++
+ PIPELINE +
++++++++++++



################################
# TEST VARIOUS TARGET SCENARIO #
################################


TARGETS
========

2
-

**Dry Run enabled**

The shell 🐚 command "false" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----

1
-

**Dry Run enabled**

The shell 🐚 command "true" ran successfully with the following output:
----
----
No change detected

4
-
WARNING: Parent target["2"] did not succeed. Skipping execution of the target["4"]

3
-

**Dry Run enabled**

The shell 🐚 command "false" exited on error (exit code 1) with the following output:
----
----

command stderr output was:
----
----

5
-

**Dry Run enabled**

The shell 🐚 command "true" ran successfully with the following output:
----
----
No change detected

ERROR: something went wrong in target "2" : "Shell command exited on error."
ERROR: something went wrong in target "3" : "Shell command exited on error."

Pipeline "Test Various target scenario" failed
Skipping due to:
	targets stage:	"something went wrong during target execution"

=============================

REPORTS:


✗ COMMAND.YAML:
	Sources:
	Target:
		✔ [1] Should be succeeding (kind: shell)
		✗ [2] Should be failling (kind: shell)
		✗ [3] Should be failling (kind: shell)
		- [4] Should be skipped (kind: shell)
		✔ [5] Should be succeeding (kind: shell)



Run Summary
===========
Pipeline(s) run:
  * Changed:	0
  * Failed:	1
  * Skipped:	0
  * Succeeded:	0
  * Total:	1
ERROR: ✗ 1 over 1 pipeline failed
ERROR: command failed
[1]    18856 exit 1     ./bin/updatecli diff --config e2e/updatecli.d/command.yaml

[olblak]

I am waiting for a final approval before merging this PR

[olblak]

After some test, I didn't catch any regression

[olblak]

Thanks @lemeurherve