Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Enhanced WorkflowTemplate and ClusterWorkflowTemplate validation to support Global Variables #2644

Merged
merged 82 commits into from
Apr 15, 2020
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
64f6403
Update cli_with_server_test.go
sarabala1979 Feb 7, 2020
4a67b77
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 13, 2020
b318144
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 14, 2020
ccc86c7
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 18, 2020
4702130
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 21, 2020
3a11f82
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 21, 2020
44d8b51
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 21, 2020
07a41f9
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 24, 2020
3ac482d
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 25, 2020
0934b2a
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 26, 2020
5b4c201
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 27, 2020
0875e7b
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Feb 28, 2020
9da394b
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 6, 2020
dff0204
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 12, 2020
3e2d0cd
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 12, 2020
1873bc1
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 12, 2020
b45c85d
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 15, 2020
b3fe528
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 15, 2020
ae62719
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 16, 2020
a49bebe
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 17, 2020
68097c1
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 18, 2020
48ece72
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 18, 2020
d151e8f
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 18, 2020
c4ebd48
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 19, 2020
b6653f6
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 19, 2020
d7b1503
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 19, 2020
d1e72e8
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 20, 2020
caf42aa
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 23, 2020
f2e2f43
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 24, 2020
a71f32d
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 25, 2020
46a86a1
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 25, 2020
0ba75d7
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 25, 2020
cd24fff
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 26, 2020
76c6b98
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 26, 2020
4aadd7a
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 30, 2020
484ff30
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 30, 2020
b123157
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 30, 2020
8c62154
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Mar 31, 2020
1777b3f
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 1, 2020
655013f
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 2, 2020
e02a98c
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 2, 2020
f5e3c8f
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 3, 2020
da26c49
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 3, 2020
bfd8602
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 6, 2020
3046680
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 6, 2020
48d3a4b
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 6, 2020
b9f9d1d
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 7, 2020
f3098d3
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 7, 2020
a7c82b7
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 7, 2020
38bc2ee
Update cluster-workflow-template-nested-template.yaml
sarabala1979 Apr 8, 2020
b9dcbf0
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 8, 2020
0e1f7f5
Improved the validatation for Workflowtemplates and ClusterWorkflowTe…
sarabala1979 Apr 8, 2020
44b053b
Merge branch 'master' into wftlint
sarabala1979 Apr 8, 2020
fde23eb
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 8, 2020
0e14398
Merge branch 'master' into wftlint
sarabala1979 Apr 9, 2020
45d0236
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 9, 2020
a0d6683
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 9, 2020
e7cc850
Merge branch 'master' into wftlint
sarabala1979 Apr 9, 2020
91b7f1b
Merge branch 'master' into wftlint
alexec Apr 10, 2020
446c94c
removed unused function
sarabala1979 Apr 10, 2020
a63ec69
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 10, 2020
6e3c552
Update workflow/validate/validate.go
sarabala1979 Apr 13, 2020
e0dda39
Merge branch 'master' into wftlint
sarabala1979 Apr 13, 2020
0145e5d
Merge branch 'master' of https://github.com/argoproj/argo
sarabala1979 Apr 13, 2020
844f53c
Update validate.go
sarabala1979 Apr 13, 2020
274e71c
Update workflow/validate/validate.go
sarabala1979 Apr 13, 2020
ea55490
Update validate.go
sarabala1979 Apr 13, 2020
f70255c
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
366d0a9
updated codeGen
sarabala1979 Apr 14, 2020
f8f294e
Merge branch 'wftlint' of https://github.com/sarabala1979/argo into w…
sarabala1979 Apr 14, 2020
71814d6
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
4a59ff2
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
0dfffb8
fixed Swagger.json
sarabala1979 Apr 14, 2020
e7edc73
Merge branch 'master' into wftlint
alexec Apr 14, 2020
15e83b5
Update go.mod
sarabala1979 Apr 14, 2020
f71abf3
Merge branch 'wftlint' of https://github.com/sarabala1979/argo into w…
sarabala1979 Apr 14, 2020
4a7ac21
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
0a78b34
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
672b23a
Merge branch 'master' into wftlint
sarabala1979 Apr 14, 2020
b0cc9be
Merge branch 'wftlint' of https://github.com/sarabala1979/argo into w…
sarabala1979 Apr 14, 2020
504ae77
Update go.sum
sarabala1979 Apr 14, 2020
118c7fd
updated Go.mod
sarabala1979 Apr 15, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 0 additions & 4 deletions pkg/apis/workflow/v1alpha1/cluster_workflow_template_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,3 @@ func (cwftmpl *ClusterWorkflowTemplate) GetResourceScope() ResourceScope {
return ResourceScopeCluster
}

// GetTemplates returns the list of templates of cluster workflow template
func (cwftmpl *ClusterWorkflowTemplate) GetTemplates() []Template {
return cwftmpl.Spec.Templates
}
1 change: 0 additions & 1 deletion pkg/apis/workflow/v1alpha1/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ type TemplateHolder interface {
GroupVersionKind() schema.GroupVersionKind
GetTemplateByName(name string) *Template
GetResourceScope() ResourceScope
GetTemplates() []Template
}

// TemplateReferenceHolder is an object that holds a reference to other templates; e.g. WorkflowStep, DAGTask, and NodeStatus
Expand Down
4 changes: 0 additions & 4 deletions pkg/apis/workflow/v1alpha1/workflow_template_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,3 @@ func (wftmpl *WorkflowTemplate) GetResourceScope() ResourceScope {
return ResourceScopeNamespaced
}

// GetTemplates returns the list of templates of workflow template
func (wftmpl *WorkflowTemplate) GetTemplates() []Template {
return wftmpl.Spec.Templates
}
4 changes: 0 additions & 4 deletions pkg/apis/workflow/v1alpha1/workflow_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -1690,10 +1690,6 @@ func (wf *Workflow) GetResourceScope() ResourceScope {
return ResourceScopeLocal
}

// GetTemplates returns the list of templates of workflow.
func (wf *Workflow) GetTemplates() []Template {
return wf.Spec.Templates
}

// NodeID creates a deterministic node ID based on a node name
func (wf *Workflow) NodeID(name string) string {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ func (cwts *ClusterWorkflowTemplateServer) CreateClusterWorkflowTemplate(ctx con

cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(nil, cwftmplGetter, req.Template)
_, err := validate.ValidateClusterWorkflowTemplate(nil, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -82,7 +82,7 @@ func (cwts *ClusterWorkflowTemplateServer) LintClusterWorkflowTemplate(ctx conte

cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(nil, cwftmplGetter, req.Template)
_, err := validate.ValidateClusterWorkflowTemplate(nil, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand All @@ -97,7 +97,7 @@ func (cwts *ClusterWorkflowTemplateServer) UpdateClusterWorkflowTemplate(ctx con
wfClient := auth.GetWfClient(ctx)
cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(nil, cwftmplGetter, req.Template)
_, err := validate.ValidateClusterWorkflowTemplate(nil, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand Down
6 changes: 3 additions & 3 deletions server/workflowtemplate/workflow_template_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ func (wts *WorkflowTemplateServer) CreateWorkflowTemplate(ctx context.Context, r

cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
_, err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -85,7 +85,7 @@ func (wts *WorkflowTemplateServer) LintWorkflowTemplate(ctx context.Context, req

cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
_, err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand All @@ -102,7 +102,7 @@ func (wts *WorkflowTemplateServer) UpdateWorkflowTemplate(ctx context.Context, r

cwftmplGetter := templateresolution.WrapClusterWorkflowTemplateInterface(wfClient.ArgoprojV1alpha1().ClusterWorkflowTemplates())

err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
_, err := validate.ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, req.Template)
if err != nil {
return nil, err
}
Expand Down
19 changes: 18 additions & 1 deletion test/e2e/testdata/cluster-workflow-template-nested-template.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
kind: ClusterWorkflowTemplate
metadata:
name: cluster-workflow-template-whalesay-template
labels:
argo-e2e: true
spec:
templates:
- name: whalesay-template
inputs:
parameters:
- name: message
container:
image: docker/whalesay
command: [cowsay]
args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
Expand All @@ -8,8 +24,9 @@ spec:
templates:
- name: whalesay-inner-template
templateRef:
name: workflow-template-whalesay-template
name: cluster-workflow-template-whalesay-template
template: whalesay-template
clusterscope: true
inputs:
parameters:
- name: message
Expand Down
2 changes: 1 addition & 1 deletion workflow/validate/lint.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func LintWorkflowTemplateFile(wftmplGetter templateresolution.WorkflowTemplateNa
return errors.Errorf(errors.CodeBadRequest, "%s failed to parse: %v", filePath, err)
}
for _, wftmpl := range workflowTemplates {
err = ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, &wftmpl)
_, err = ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, &wftmpl)
if err != nil {
return errors.Errorf(errors.CodeBadRequest, "%s: %s", filePath, err.Error())
}
Expand Down
35 changes: 19 additions & 16 deletions workflow/validate/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ type ValidateOpts struct {
// types of executors. For example, the inability of kubelet/k8s executors to copy artifacts
// out of the base image layer. If unspecified, will use docker executor validation
ContainerRuntimeExecutor string

// IgnoreEntrypoint indicates to skip/ignore the EntryPoint validation on workflow spec.
// Entrypoint is optional for WorkflowTemplate and ClusterWorkflowTemplate
IgnoreEntrypoint bool
}

// templateValidationCtx is the context for validating a workflow spec
Expand Down Expand Up @@ -134,7 +138,7 @@ func ValidateWorkflow(wftmplGetter templateresolution.WorkflowTemplateNamespaced
ctx.globalParams[common.GlobalVarWorkflowPriority] = strconv.Itoa(int(*wf.Spec.Priority))
}

if wf.Spec.Entrypoint == "" {
if !opts.IgnoreEntrypoint && wf.Spec.Entrypoint == "" {
return nil, errors.New(errors.CodeBadRequest, "spec.entrypoint is required")
}

Expand Down Expand Up @@ -166,9 +170,11 @@ func ValidateWorkflow(wftmplGetter templateresolution.WorkflowTemplateNamespaced
}
}

_, err = ctx.validateTemplateHolder(&wfv1.WorkflowStep{Template: wf.Spec.Entrypoint}, tmplCtx, &wf.Spec.Arguments, map[string]interface{}{})
if err != nil {
return nil, err
if !opts.IgnoreEntrypoint {
_, err = ctx.validateTemplateHolder(&wfv1.WorkflowStep{Template: wf.Spec.Entrypoint}, tmplCtx, &wf.Spec.Arguments, map[string]interface{}{})
if err != nil {
return nil, err
}
}
if wf.Spec.OnExit != "" {
// now when validating onExit, {{workflow.status}} is now available as a global
Expand Down Expand Up @@ -198,19 +204,16 @@ func ValidateWorkflow(wftmplGetter templateresolution.WorkflowTemplateNamespaced
return wfConditions, nil
}

// ValidateWorkflow accepts a workflow template and performs validation against it.
func ValidateWorkflowTemplate(wftmplGetter templateresolution.WorkflowTemplateNamespacedGetter, cwftmplGetter templateresolution.ClusterWorkflowTemplateGetter, wftmpl wfv1.TemplateHolder) error {
ctx := newTemplateValidationCtx(nil, ValidateOpts{})
tmplCtx := templateresolution.NewContext(wftmplGetter, cwftmplGetter, wftmpl, nil)
// ValidateWorkflowTemplate accepts a workflow template and performs validation against it.
func ValidateWorkflowTemplate(wftmplGetter templateresolution.WorkflowTemplateNamespacedGetter, cwftmplGetter templateresolution.ClusterWorkflowTemplateGetter, wftmpl *wfv1.WorkflowTemplate) (*wfv1.WorkflowConditions, error) {
wf := common.ConvertWorkflowTemplateToWorkflow(wftmpl)
return ValidateWorkflow(wftmplGetter, cwftmplGetter, wf, ValidateOpts{IgnoreEntrypoint: wf.Spec.Entrypoint == ""})
sarabala1979 marked this conversation as resolved.
Show resolved Hide resolved
}

// Check if all templates can be resolved.
for _, template := range wftmpl.GetTemplates() {
_, err := ctx.validateTemplateHolder(&wfv1.WorkflowStep{Template: template.Name}, tmplCtx, &FakeArguments{}, map[string]interface{}{})
if err != nil {
return errors.Errorf(errors.CodeBadRequest, "templates.%s %s", template.Name, err.Error())
}
}
return nil
// ValidateClusterWorkflowTemplate accepts a cluster workflow template and performs validation against it.
func ValidateClusterWorkflowTemplate(wftmplGetter templateresolution.WorkflowTemplateNamespacedGetter, cwftmplGetter templateresolution.ClusterWorkflowTemplateGetter, cwftmpl *wfv1.ClusterWorkflowTemplate) (*wfv1.WorkflowConditions, error) {
wf := common.ConvertClusterWorkflowTemplateToWorkflow(cwftmpl)
return ValidateWorkflow(wftmplGetter, cwftmplGetter, wf, ValidateOpts{IgnoreEntrypoint: wf.Spec.Entrypoint == ""})
}

// ValidateCronWorkflow validates a CronWorkflow
Expand Down
3 changes: 2 additions & 1 deletion workflow/validate/validate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ func validate(yamlStr string) (*wfv1.WorkflowConditions, error) {
// its validation result.
func validateWorkflowTemplate(yamlStr string) error {
wftmpl := unmarshalWftmpl(yamlStr)
return ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, wftmpl)
_, err := ValidateWorkflowTemplate(wftmplGetter, cwftmplGetter, wftmpl)
return err
}

func unmarshalWf(yamlStr string) *wfv1.Workflow {
Expand Down