workflow task node #1866

STRRL · 2021-05-20T03:59:19Z

Signed-off-by: STRRL str_ruiling@outlook.com

What problem does this PR solve?

This PR implements the task node for workflow.

What is changed and how does it work?

this PR adds two new modules in /pkg/expr and /pkg/workflow/task
expr use github.com/antonmedv/expr for evaluating expression
/pkg/workflow/task will create Pod by given spec, and collect information during execution, like stdout and exitcode
the rest of task reconcile is very similar to parallel node reconcile, but with conditions

Checklist

Tests

Unit test
E2E test
Manual test (add detailed scripts or steps below)
No code

Side effects

Breaking backward compatibility

Related changes

Need to update the documentation

Does this PR introduce a user-facing change?

NONE

Signed-off-by: STRRL <str_ruiling@outlook.com>

pkg/workflow/task/collector/exit_code.go

codecov-commenter · 2021-05-20T04:08:51Z

Codecov Report

Merging #1866 (2ebfcff) into master (18ad845) will decrease coverage by 5.83%.
The diff coverage is 45.94%.

@@            Coverage Diff             @@
##           master    #1866      +/-   ##
==========================================
- Coverage   39.47%   33.63%   -5.84%     
==========================================
  Files          17      121     +104     
  Lines         608     7283    +6675     
==========================================
+ Hits          240     2450    +2210     
- Misses        337     4624    +4287     
- Partials       31      209     +178

Impacted Files	Coverage Δ
api/v1alpha1/awschaos_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/dnschaos_type.go	`0.00% <0.00%> (ø)`
api/v1alpha1/dnschaos_webhook.go	`0.00% <0.00%> (ø)`
api/v1alpha1/gcpchaos_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/httpchaos_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/httpchaos_webhook.go	`0.00% <0.00%> (ø)`
api/v1alpha1/iochaos_types.go	`0.00% <0.00%> (-32.56%)`	⬇️
api/v1alpha1/jvmchaos_webhook.go	`0.00% <0.00%> (ø)`
api/v1alpha1/kernelchaos_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/networkchaos_types.go	`0.00% <0.00%> (-14.44%)`	⬇️
... and 185 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae11cf7...2ebfcff. Read the comment docs.

Signed-off-by: STRRL <str_ruiling@outlook.com>

compliable issue with kubernetes 1.18, detail: kubernetes-sigs/controller-tools#480 Signed-off-by: STRRL <str_ruiling@outlook.com>

Signed-off-by: STRRL <str_ruiling@outlook.com>

This reverts commit 8e05196. Signed-off-by: STRRL <str_ruiling@outlook.com>

Signed-off-by: STRRL <str_ruiling@outlook.com>

STRRL · 2021-06-11T03:17:56Z

Weird, the integration test fails. I am looking on that.

AsterNighT · 2021-06-11T03:24:36Z

Weird, the integration test fails. I am looking on that.

The CustomResourceDefinition "workflownodes.chaos-mesh.org" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

STRRL · 2021-06-11T03:29:43Z

Weird, the integration test fails. I am looking on that.

The CustomResourceDefinition "workflownodes.chaos-mesh.org" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I could not reproduce on my local cluster, both k8s v1.15.x and v1.20.x, and both crd v1beta1 and crd v1. 😢~~Still finding why.~~

I think I found the reason, kubectl wants to create an annotation called last-applied-configuration which contains the full content of the CRD. We should use kubectl create instead of kubectl apply

I am going to fix it.

apply will create a huge annotaion which exceed the limit of size of annotation Signed-off-by: STRRL <str_ruiling@outlook.com>

api/v1alpha1/workflow_types.go

Signed-off-by: STRRL <str_ruiling@outlook.com>

YangKeao

Except the name, LGTM.

The fields will be renamed in another PR, right?

ti-chi-bot · 2021-06-16T03:30:43Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

AsterNighT
YangKeao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

STRRL · 2021-06-16T03:54:39Z

The fields will be renamed in another PR, right?

YES!

STRRL · 2021-06-16T03:54:49Z

/merge

ti-chi-bot · 2021-06-16T03:54:51Z

This pull request has been accepted and is ready to merge.

Commit hash: 06af0af

STRRL · 2021-06-16T04:52:20Z

/run-e2e-tests

Signed-off-by: STRRL <str_ruiling@outlook.com>

cwen0 · 2021-06-16T07:23:50Z

/merge

ti-chi-bot · 2021-06-16T07:23:51Z

This pull request has been accepted and is ready to merge.

Commit hash: b0d2c5f

* feat: conditional branches in workflownode Signed-off-by: STRRL <str_ruiling@outlook.com> * feat: reconciler of task node Signed-off-by: STRRL <str_ruiling@outlook.com> * feat: setup one container for task node Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: address the comments by muse-dev Signed-off-by: STRRL <str_ruiling@outlook.com> * chore: cleanup duplicated codes Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: bump controller-gen version compliable issue with kubernetes 1.18, detail: kubernetes-sigs/controller-tools#480 Signed-off-by: STRRL <str_ruiling@outlook.com> * chore: bump the version of controller-gen to v0.4.1 Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: restrict kubernetes version with replace Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: also restrict version for e2e-test Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: also bump the verison of controller-runtime Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: fallback to v1beta1 Signed-off-by: STRRL <str_ruiling@outlook.com> * Revert "fix: also bump the verison of controller-runtime" This reverts commit 8e05196. Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: fill the conditional branches if nil Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: permission of custom task about creating pod Signed-off-by: STRRL <str_ruiling@outlook.com> * feat: use v1 instead of v1beta1 with CRD Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: NOOP when pods exactly same Signed-off-by: STRRL <str_ruiling@outlook.com> * refactor: refactor the collectors Signed-off-by: STRRL <str_ruiling@outlook.com> * feat: finish the rest of task nodes Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: check before sync nodes in task Signed-off-by: STRRL <str_ruiling@outlook.com> * chore: make linters happier Signed-off-by: STRRL <str_ruiling@outlook.com> * chore: make linters happier Signed-off-by: STRRL <str_ruiling@outlook.com> * Update pkg/workflow/controllers/task_reconciler.go Signed-off-by: STRRL <str_ruiling@outlook.com> Co-authored-by: AsterNighT <klxjt99@outlook.com> * fix: use create instead of apply apply will create a huge annotaion which exceed the limit of size of annotation Signed-off-by: STRRL <str_ruiling@outlook.com> * chore: update the comments for workflow template Signed-off-by: STRRL <str_ruiling@outlook.com> * fix: crd installation in e2e test Signed-off-by: STRRL <str_ruiling@outlook.com> Co-authored-by: AsterNighT <klxjt99@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Signed-off-by: root <root@vm3.jisiqhvhuzaupp2x5y5vff1eff.fx.internal.cloudapp.net>

STRRL added 2 commits May 18, 2021 20:06

feat: conditional branches in workflownode

51f87b3

Signed-off-by: STRRL <str_ruiling@outlook.com>

feat: reconciler of task node

922bd51

Signed-off-by: STRRL <str_ruiling@outlook.com>

ti-chi-bot added do-not-merge/work-in-progress needs-rebase labels May 20, 2021

ti-chi-bot requested review from Colstuwjx and fewdan May 20, 2021 03:59

ti-chi-bot added the size/XXL label May 20, 2021

Merge remote-tracking branch 'upstream/master' into workflow-task

85ee9c0

ti-chi-bot removed the needs-rebase label May 20, 2021

sonatype-lift bot reviewed May 20, 2021

View reviewed changes

pkg/workflow/task/collector/exit_code.go Outdated Show resolved Hide resolved

STRRL added 5 commits May 21, 2021 00:39

feat: setup one container for task node

554699c

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: address the comments by muse-dev

1116a90

Signed-off-by: STRRL <str_ruiling@outlook.com>

Merge remote-tracking branch 'upstream/master' into workflow-task

f8887cd

chore: cleanup duplicated codes

b402245

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: bump controller-gen version

282e06d

compliable issue with kubernetes 1.18, detail: kubernetes-sigs/controller-tools#480 Signed-off-by: STRRL <str_ruiling@outlook.com>

STRRL force-pushed the workflow-task branch from 0f9597f to 282e06d Compare May 24, 2021 05:01

STRRL mentioned this pull request May 24, 2021

chore: bump the version of controller-gen to v0.4.1 #1880

Merged

6 tasks

STRRL added 12 commits May 24, 2021 16:37

chore: bump the version of controller-gen to v0.4.1

3f99413

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: restrict kubernetes version with replace

ce8cb99

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: also restrict version for e2e-test

f33418f

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: also bump the verison of controller-runtime

8e05196

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: fallback to v1beta1

57ee991

Signed-off-by: STRRL <str_ruiling@outlook.com>

Revert "fix: also bump the verison of controller-runtime"

72a40ba

This reverts commit 8e05196. Signed-off-by: STRRL <str_ruiling@outlook.com>

Merge branch 'bump-controller-gen-version' into workflow-task

90ee1e3

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: fill the conditional branches if nil

20cbd02

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: permission of custom task about creating pod

3b280ac

Signed-off-by: STRRL <str_ruiling@outlook.com>

feat: use v1 instead of v1beta1 with CRD

c39b216

Signed-off-by: STRRL <str_ruiling@outlook.com>

fix: NOOP when pods exactly same

b36bf43

Signed-off-by: STRRL <str_ruiling@outlook.com>

refactor: refactor the collectors

aac38f4

Signed-off-by: STRRL <str_ruiling@outlook.com>

ti-chi-bot removed the needs-rebase label Jun 11, 2021

fix: use create instead of apply

001b6f6

apply will create a huge annotaion which exceed the limit of size of annotation Signed-off-by: STRRL <str_ruiling@outlook.com>

cwen0 reviewed Jun 15, 2021

View reviewed changes

api/v1alpha1/workflow_types.go Show resolved Hide resolved

chore: update the comments for workflow template

06af0af

Signed-off-by: STRRL <str_ruiling@outlook.com>

YangKeao approved these changes Jun 16, 2021

View reviewed changes

ti-chi-bot added status/LGT2 and removed status/LGT1 labels Jun 16, 2021

ti-chi-bot added the status/can-merge label Jun 16, 2021

Merge branch 'master' into workflow-task

e147614

fix: crd installation in e2e test

b0d2c5f

Signed-off-by: STRRL <str_ruiling@outlook.com>

ti-chi-bot removed the status/can-merge label Jun 16, 2021

ti-chi-bot added the status/can-merge label Jun 16, 2021

ti-chi-bot added 3 commits June 16, 2021 15:28

Merge branch 'master' into workflow-task

0bd4eb9

Merge branch 'master' into workflow-task

2c72047

Merge branch 'master' into workflow-task

2ebfcff

ti-chi-bot merged commit c8be4ad into chaos-mesh:master Jun 16, 2021

STRRL mentioned this pull request Jun 17, 2021

rename confusion names in workflow definition and status #1984

Merged

5 tasks

STRRL mentioned this pull request Jun 24, 2021

use intall.sh to reinstall Chaos Mesh #2030

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workflow task node #1866

workflow task node #1866

STRRL commented May 20, 2021 •

edited

codecov-commenter commented May 20, 2021 •

edited

STRRL commented Jun 11, 2021

AsterNighT commented Jun 11, 2021

STRRL commented Jun 11, 2021 •

edited

YangKeao left a comment

ti-chi-bot commented Jun 16, 2021

STRRL commented Jun 16, 2021

STRRL commented Jun 16, 2021

ti-chi-bot commented Jun 16, 2021

STRRL commented Jun 16, 2021

cwen0 commented Jun 16, 2021

ti-chi-bot commented Jun 16, 2021

workflow task node #1866

workflow task node #1866

Conversation

STRRL commented May 20, 2021 • edited

What problem does this PR solve?

What is changed and how does it work?

Checklist

Does this PR introduce a user-facing change?

codecov-commenter commented May 20, 2021 • edited

Codecov Report

STRRL commented Jun 11, 2021

AsterNighT commented Jun 11, 2021

STRRL commented Jun 11, 2021 • edited

YangKeao left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Jun 16, 2021

STRRL commented Jun 16, 2021

STRRL commented Jun 16, 2021

ti-chi-bot commented Jun 16, 2021

STRRL commented Jun 16, 2021

cwen0 commented Jun 16, 2021

ti-chi-bot commented Jun 16, 2021

STRRL commented May 20, 2021 •

edited

codecov-commenter commented May 20, 2021 •

edited

STRRL commented Jun 11, 2021 •

edited