Setting executor (wait) container resources in the WorkflowTemplate is counterintuitive/undocumented

[diversario]

I want to configure the default wait container to be very small, and adjust its resources, if necessary, from the WorkflowTemplates.

I have tried these approaches, after reading the docs:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  podSpecPatch: |-
    {"containers": [{"name": "wait", "resources": {"requests": {"cpu": "51m", "memory": "128Mi"}, "limits": {"cpu": "51m", "memory": "128Mi"}}}]}
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  podSpecPatch: |-
    containers:
    - name: wait
      resources:
        requests:
          cpu: "51m"
          memory: "128Mi"
        limits:
          cpu: "51m"
          memory: "128Mi"

but the wait container was unaffected. The wait container is not defined in the workflow template, so the spec-level podSpecPatch seems to be the only place for the patch, but this worked:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
  templates:
  - name: image
    podSpecPatch: |
      initContainers:
      - name: init
        resources:
          requests:
            cpu: "53m"
            memory: "128Mi"
          limits:
            cpu: "54m"
            memory: "128Mi"
      containers:
      - name: wait
        resources:
          requests:
            cpu: "55m"
            memory: "129Mi"
          limits:
            cpu: "56m"
            memory: "129Mi"
    container:
      image: moby/buildkit:v0.11.6-rootless
    # the rest of the template

even though according to the documentation, the correct place is .spec.podSpecPatch at the root of the WFT.

[jswxstw]

even though according to the documentation, the correct place is .spec.podSpecPatch at the root of the WFT.

.spec.podSpecPatch in the documentation is actually workflow-level podSpecPatch
.spec.templates[0].podSpecPatch is template-level podSpecPatch

argo-workflows/workflow/controller/workflowpod.go

Lines 362 to 383 in 7286d49

	if woc.hasPodSpecPatch(tmpl) {
	tmpl.PodSpecPatch, err = util.PodSpecPatchMerge(woc.wf, tmpl)
	if err != nil {
	return nil, errors.Wrap(err, "", "Failed to merge the workflow PodSpecPatch with the template PodSpecPatch due to invalid format")
	}
	// Final substitution for workflow level PodSpecPatch
	localParams := make(map[string]string)
	if tmpl.IsPodType() {
	localParams[common.LocalVarPodName] = pod.Name
	}
	tmpl, err := common.ProcessArgs(tmpl, &wfv1.Arguments{}, woc.globalParams, localParams, false, woc.wf.Namespace, woc.controller.configMapInformer.GetIndexer())
	if err != nil {
	return nil, errors.Wrap(err, "", "Failed to substitute the PodSpecPatch variables")
	}
	patchedPodSpec, err := util.ApplyPodSpecPatch(pod.Spec, tmpl.PodSpecPatch)
	if err != nil {
	return nil, errors.Wrap(err, "", "Error applying PodSpecPatch")
	}
	pod.Spec = *patchedPodSpec
	}

Unfortunately, workflow-level podSpecPatch in WorkflowTemplate can not take effect.

[diversario]

Ah, I see. And, seems like it's the first WF in the WFT that should have the podSpecPatch for those default (init, wait) containers?..

[jswxstw]

Are you using workflowTemplateRef submitting WFT with workflow-level podSpecPatch?

[diversario]

Yes, these WFTs are used by the argo events sensors like:

triggers:
- template:
    argoWorkflow:
      operation: submit
      source:
        resource:
          apiVersion: argoproj.io/v1alpha1
          kind: Workflow
          spec:
            workflowTemplateRef:
              name: stuff

[jswxstw]

Unfortunately, workflow-level podSpecPatch in WorkflowTemplate can not take effect.

It seems like a bug when using workflowTemplateRef to submit a new workflow with WFT.
PodSpecPatch in woc.wf will be applied, but woc.wf should not be used in execution logic. woc.execWf.Spec should be used instead.

[jswxstw]

It will be fixed with this PR: #12476