Pods created by Argo Workflows don't have the desired labels defined by the podMetadata field

[alexglatt]

Summary

I need to submit Argo Workflows to create pods that have specific labels. However, most of the time when I use the podMetadata field in the WorkflowSpec, the pods don't have the defined labels. In some rare submission attempts, the created pods do have the specified labels but when I submit the exact same Workflow once again, the created pods don't have them again.
When I submit a Workflow with multiple steps, the created pods for each step randomly have the labels. Sometimes one pod has it, sometimes two, sometimes all of them, but most of the time no created pod has the label.

I actually even want to define the pod label using the podMetadata Field in a WorkflowTemplate. However, when I specify it in the WorkflowTemplate, the created pods using a Workflow that references this WorkflowTemplate never get the labels.

For now, it would work for me if I can't specify the labels in the WorkflowTemplate. However, I need to be sure that the pods definitely receive their labels when I specify them in the Workflow.

Argo Version: v3.1.14
Kubectl Version: v1.22.2

Diagnostics

Bug can be reproduced easily with the HelloWorld example:
(For me this example produces 3 pods where all 3 almost never receive the label. Most of the time even no pod gets the label. Especially when submitting the Workflow multiple times.)

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay
  
  podMetadata:
    labels:
      test: label
  
  templates:
  - name: whalesay
    steps:                          
      - - name: whalesay-english
          template: whalesay-english
        - name: whalesay-spanish
          template: whalesay-spanish
      - - name: whalesay-french
          template: whalesay-french
          
  - name: whalesay-english 
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]
  
  - name: whalesay-spanish
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hola mundo"]
      
  - name: whalesay-french
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["bonjour le monde"]

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

I actually even want to define the pod label using the podMetadata Field in a WorkflowTemplate. However, when I specify it in the WorkflowTemplate, the created pods using a Workflow that references this WorkflowTemplate never get the labels.

If you refer the workflowtemplate using templateRef, spec.podMetadata will be applied because it is in spec scope. if you use workflowTemplateRef, it will be applied.

[sarabala1979]

@alexglatt I tried in my local env multiple time. it works. Can you provide me the controller log when it didn't have it?

[alexglatt]

Thank you for your response @sarabala1979 !

Unfortunately, the labels also don't get included from the Workflow Template if I put them in the WorkflowTemplate spec.podMetadata and reference it with the workflowTemplateRef... Also it seems, that I can not use steps when using workflowTemplateRef.
However, since not even the labels defined in the Workflow are included in the pods, this might be a general problem in my environment.

Do you need the argo logs? Those seem to work just fine, no matter if the labels get included or not. They look as intended in the hello-world workflow:
(In case you need the verbose logs or the kubectl describe pod output, I can provide you those as well.)

hello-world-wq6pf-4223406590:  ____________
hello-world-wq6pf-4223406590: < hola mundo >
hello-world-wq6pf-4223406590:  ------------
hello-world-wq6pf-4223406590:     \
hello-world-wq6pf-4223406590:      \
hello-world-wq6pf-4223406590:       \
hello-world-wq6pf-4223406590:                     ##        .
hello-world-wq6pf-4223406590:               ## ## ##       ==
hello-world-wq6pf-4223406590:            ## ## ## ##      ===
hello-world-wq6pf-4223406590:        /""""""""""""""""___/ ===
hello-world-wq6pf-4223406590:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
hello-world-wq6pf-4223406590:        \______ o          __/
hello-world-wq6pf-4223406590:         \    \        __/
hello-world-wq6pf-4223406590:           \____\______/
hello-world-wq6pf-3486525838:  _____________
hello-world-wq6pf-3486525838: < hello world >
hello-world-wq6pf-3486525838:  -------------
hello-world-wq6pf-3486525838:     \
hello-world-wq6pf-3486525838:      \
hello-world-wq6pf-3486525838:       \
hello-world-wq6pf-3486525838:                     ##        .
hello-world-wq6pf-3486525838:               ## ## ##       ==
hello-world-wq6pf-3486525838:            ## ## ## ##      ===
hello-world-wq6pf-3486525838:        /""""""""""""""""___/ ===
hello-world-wq6pf-3486525838:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
hello-world-wq6pf-3486525838:        \______ o          __/
hello-world-wq6pf-3486525838:         \    \        __/
hello-world-wq6pf-3486525838:           \____\______/
hello-world-wq6pf-2131079227:  __________________
hello-world-wq6pf-2131079227: < bonjour le monde >
hello-world-wq6pf-2131079227:  ------------------
hello-world-wq6pf-2131079227:     \
hello-world-wq6pf-2131079227:      \
hello-world-wq6pf-2131079227:       \
hello-world-wq6pf-2131079227:                     ##        .
hello-world-wq6pf-2131079227:               ## ## ##       ==
hello-world-wq6pf-2131079227:            ## ## ## ##      ===
hello-world-wq6pf-2131079227:        /""""""""""""""""___/ ===
hello-world-wq6pf-2131079227:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
hello-world-wq6pf-2131079227:        \______ o          __/
hello-world-wq6pf-2131079227:         \    \        __/
hello-world-wq6pf-2131079227:           \____\______/

[sarabala1979]

@alexglatt

Unfortunately, the labels also don't get included from the Workflow Template if I put them in the WorkflowTemplate spec.podMetadata and reference it with the workflowTemplateRef...

WorkflowTemplate spec.podMetadata should include in pod if your workflow is using workflowTemplateRef. If you are not seeing. Please provide your workflowtemplate and workflow.

Also it seems, that I can not use steps when using workflowTemplateRef.

If workflow is referring workflowTemplateRef, you can't use any other templates in workflow. workflowTemplateRef assume entire workflow is defined as workflowtemplate

However, since not even the labels defined in the Workflow are included in the pods, this might be a general problem in my environment.

Do you have any MutatingAdmissionWebhook on your env to remove the labels?

[sajjanjyothi]

seems like pod's labels are hardcoded in workflowpod.go, Probably we should consider appending workflow labels to pod's labels along with hardcoded labels from controller.
Below code may help

diff --git a/workflow/controller/workflowpod.go b/workflow/controller/workflowpod.go
index 112831a79..11c9c5e61 100644
--- a/workflow/controller/workflowpod.go
+++ b/workflow/controller/workflowpod.go
@@ -191,14 +191,20 @@ func (woc *wfOperationCtx) createWorkflowPod(ctx context.Context, nodeName strin
                        activeDeadlineSeconds = tmplActiveDeadlineSeconds
                }
        }
+
+       //append to the existing workflow labels
+       for k, v := range map[string]string{
+               common.LabelKeyWorkflow:  woc.wf.ObjectMeta.Name, // Allows filtering by pods related to specific workflow
+               common.LabelKeyCompleted: "false",                // Allows filtering by incomplete workflow pods
+       } {
+               woc.wf.ObjectMeta.Labels[k] = v
+       }
+
        pod := &apiv1.Pod{
                ObjectMeta: metav1.ObjectMeta{
                        Name:      util.PodName(woc.wf.Name, nodeName, tmpl.Name, nodeID),
                        Namespace: woc.wf.ObjectMeta.Namespace,
-                       Labels: map[string]string{
-                               common.LabelKeyWorkflow:  woc.wf.ObjectMeta.Name, // Allows filtering by pods related to specific workflow
-                               common.LabelKeyCompleted: "false",                // Allows filtering by incomplete workflow pods
-                       },
+                       Labels:    woc.wf.ObjectMeta.Labels,
                        Annotations: map[string]string{
                                common.AnnotationKeyNodeName: nodeName,
                                common.AnnotationKeyNodeID:   nodeID,

[sarabala1979]

@alexglatt there is downline function will add all podmetadata

// addMetadata applies metadata specified in the template
func (woc *wfOperationCtx) addMetadata(pod *apiv1.Pod, tmpl *wfv1.Template) {
	if woc.execWf.Spec.PodMetadata != nil {
		// add workflow-level pod annotations and labels
		for k, v := range woc.execWf.Spec.PodMetadata.Annotations {
			pod.ObjectMeta.Annotations[k] = v
		}
		for k, v := range woc.execWf.Spec.PodMetadata.Labels {
			pod.ObjectMeta.Labels[k] = v
		}
	}

	for k, v := range tmpl.Metadata.Annotations {
		pod.ObjectMeta.Annotations[k] = v
	}
	for k, v := range tmpl.Metadata.Labels {
		pod.ObjectMeta.Labels[k] = v
	}
}

[alexglatt]

Thank you for your answers!
Is there a way to invoke this function from my Workflow or is it normally already invoked?
I'll try to find if there are some Mutating Admission Webhooks that might remove the labels.

[sarabala1979]

@alexglatt This function will always be invoked during pod creation

[sarabala1979]

@alexglatt Can you check you have any MutatingWebhookConfiguration that removes the labels?

[alexglatt]

Hi @sarabala1979

I couldn't find if there are any Mutating/Validating Admission Webhooks that might change/remove the Labels.

However, I found something different that looks like it could be the problem.
These are the Logs in the workflow controller when I submit the Hello World example from above:

time="2021-11-10T14:03:40.703Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.706Z" level=info msg="Get configmaps 200"
time="2021-11-10T14:03:40.706Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo/#"
time="2021-11-10T14:03:40.706Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.707Z" level=info msg="Steps node hello-world-wq4cn initialized Running" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.707Z" level=info msg="StepGroup node hello-world-wq4cn-1156917824 initialized Running" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.707Z" level=info msg="Pod node hello-world-wq4cn-1923223739 initialized Pending" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.715Z" level=info msg="Create events 201"
time="2021-11-10T14:03:40.724Z" level=info msg="Create pods 409"
time="2021-11-10T14:03:40.724Z" level=info msg="Failed pod hello-world-wq4cn[0].whalesay-english (hello-world-wq4cn-1923223739) creation: already exists" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.724Z" level=info msg="Pod node hello-world-wq4cn-253801051 initialized Pending" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.765Z" level=info msg="Create pods 201"
time="2021-11-10T14:03:40.765Z" level=info msg="Created pod: hello-world-wq4cn[0].whalesay-spanish (hello-world-wq4cn-253801051)" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.765Z" level=info msg="Pod node hello-world-wq4cn-1456962725 initialized Pending" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.808Z" level=info msg="Create pods 409"
time="2021-11-10T14:03:40.808Z" level=info msg="Failed pod hello-world-wq4cn[0].whalesay-french (hello-world-wq4cn-1456962725) creation: already exists" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.808Z" level=info msg="Workflow step group node hello-world-wq4cn-1156917824 not yet completed" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.808Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.809Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-wq4cn
time="2021-11-10T14:03:40.837Z" level=info msg="Update workflows 200"
time="2021-11-10T14:03:40.837Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=180675 workflow=hello-world-wq4cn

For the both Pods where the creation failed because it "already exists", the Label is left out of the Pod. For the one Pod where the creation succeeded without problems, the Label was added to the Pod. This varies from submission to submission. Most of the times all three Pod creations fail at first try because they "already exist" (although they don't already exist in the environment) and sometimes one or more pods in the submission succeed without this problem and receive the intended Label.

The Logs of the failing Pods (in the case from above "whalesay-english" and "whalesay-french") look like this:

time="2021-11-10T14:03:42.311Z" level=info msg="Starting Workflow Executor" version=latest+d19d4ee.dirty
time="2021-11-10T14:03:42.315Z" level=info msg="Creating a docker executor"
time="2021-11-10T14:03:42.315Z" level=info msg="Executor (version: latest+d19d4ee.dirty, build_date: 2021-02-18T16:28:07Z) initialized (pod: argo/hello-world-wq4cn-1456962725) with template:\n{\"name\":\"whalesay-french\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"bonjour le monde\"],\"resources\":{}}}"
time="2021-11-10T14:03:42.315Z" level=info msg="Waiting on main container"
time="2021-11-10T14:03:48.426Z" level=info msg="main container started with container ID: <cointainerID>"
time="2021-11-10T14:03:48.426Z" level=info msg="Starting annotations monitor"
time="2021-11-10T14:03:48.431Z" level=info msg="docker wait <cointainerID>"
time="2021-11-10T14:03:48.431Z" level=info msg="Starting deadline monitor"
time="2021-11-10T14:03:48.478Z" level=info msg="Main container completed"
time="2021-11-10T14:03:48.478Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-11-10T14:03:48.478Z" level=info msg="Capturing script exit code"
time="2021-11-10T14:03:48.479Z" level=info msg="Annotations monitor stopped"
time="2021-11-10T14:03:48.529Z" level=info msg="No output parameters"
time="2021-11-10T14:03:48.529Z" level=info msg="No output artifacts"
time="2021-11-10T14:03:48.529Z" level=info msg="Annotating pod with output"
time="2021-11-10T14:03:48.587Z" level=info msg="Killing sidecars"
time="2021-11-10T14:03:48.590Z" level=info msg="Alloc=6439 TotalAlloc=15036 Sys=70848 NumGC=5 Goroutines=9"

It's a lot shorter than the Logs from the Pods where the Label is added and the Pod is created on first try. Already the first line of the Log is different for the successful Pods. There it is:

time="2021-11-10T14:03:42.299Z" level=info msg="Starting Workflow Executor" executorType= version=v3.2.3

Could the problem also be here?
If needed I can also provide the Logs of a successful Pod creation with the Label.

Thank you in advance.

[alexglatt]

In some example yaml files from the Argo Documentation Website, I found that the 'template' field also has a 'metadata' field that is supposed to set annotations and labels for Pods.
When I declare my desired metadata there instead of the 'podMetadata' field in 'spec', all the labels and annotations are successfully included in all the Pods.
This is also possible in the WorkflowTemplate. Therefore, it additionally solves my problem that the 'podMetadata' from the WorkflowTemplate never worked at all.

Thank you for your help!