quick start fails out of the box due to RBAC error

[cardoe]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Able to run the quick start CLI with success.

My environment was a macOS M2 machine running Docker Desktop with kind. The steps are straight from https://argoproj.github.io/argo-workflows/quick-start/

kind create cluster
# cluster created with 1.27.3
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.2/install.yaml
kubectl patch deployment \
  argo-server \
  --namespace argo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
  "server",
  "--auth-mode=server"
]}]'

At this point I ran:

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

And the output was:

Name:                hello-world-g2csp
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Error
Message:             Error (exit code 1): pods "hello-world-g2csp" is forbidden: User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"
Conditions:
 PodRunning          False
 Completed           True
Created:             Wed Dec 20 10:25:35 -0600 (9 minutes ago)
Started:             Wed Dec 20 10:25:35 -0600 (9 minutes ago)
Finished:            Wed Dec 20 10:26:05 -0600 (9 minutes ago)
Duration:            30 seconds
Progress:            0/1
ResourcesDuration:   13s*(1 cpu),13s*(100Mi memory)

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-g2csp  whalesay  hello-world-g2csp  21s       Error (exit code 1): pods "hello-world-g2csp" is forbidden: User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"

Version

v3.5.2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
  annotations:
    workflows.argoproj.io/description: |
      This is a simple hello world example.
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

Logs from the workflow controller

time="2023-12-20T16:25:35.731Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.736Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-g2csp
time="2023-12-20T16:25:35.736Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.736Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.736Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.736Z" level=info msg="Pod node hello-world-g2csp initialized Pending" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.745Z" level=info msg="Created pod: hello-world-g2csp (hello-world-g2csp)" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.746Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.746Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:35.752Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=2676 workflow=hello-world-g2csp
time="2023-12-20T16:25:45.751Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:45.752Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-g2csp
time="2023-12-20T16:25:45.752Z" level=info msg="node changed" namespace=argo new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=hello-world-g2csp old.message= old.phase=Pending old.progress=0/1 workflow=hello-world-g2csp
time="2023-12-20T16:25:45.752Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:45.752Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:45.765Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=2703 workflow=hello-world-g2csp
time="2023-12-20T16:25:55.768Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:55.768Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-g2csp
time="2023-12-20T16:25:55.768Z" level=info msg="node unchanged" namespace=argo nodeID=hello-world-g2csp workflow=hello-world-g2csp
time="2023-12-20T16:25:55.769Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:25:55.769Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.933Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.933Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="Pod failed: Error (exit code 1): pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" displayName=hello-world-g2csp namespace=argo pod=hello-world-g2csp templateName=whalesay workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1): pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" new.phase=Error new.progress=0/1 nodeID=hello-world-g2csp old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="Updated message  -> Error (exit code 1): pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.934Z" level=info msg="Marking workflow completed" namespace=argo workflow=hello-world-g2csp
time="2023-12-20T16:26:05.941Z" level=info msg="cleaning up pod" action=deletePod key=argo/hello-world-g2csp-1340600742-agent/deletePod
time="2023-12-20T16:26:05.951Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=2745 workflow=hello-world-g2csp
time="2023-12-20T16:26:05.959Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/hello-world-g2csp/labelPodCompleted

Logs from in your workflow's wait container

time="2023-12-20T16:25:56.956Z" level=info msg="No output parameters"
time="2023-12-20T16:25:56.956Z" level=info msg="No output artifacts"
time="2023-12-20T16:25:56.967Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-12-20T16:25:56.968Z" level=warning msg="Non-transient error: pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-20T16:25:56.968Z" level=error msg="executor error: pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-20T16:25:56.968Z" level=info msg="Alloc=7218 TotalAlloc=13045 Sys=24677 NumGC=4 Goroutines=8"
time="2023-12-20T16:25:56.969Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-12-20T16:25:56.969Z" level=warning msg="Non-transient error: pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-20T16:25:56.969Z" level=error msg="executor error: pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-20T16:25:56.970Z" level=fatal msg="pods \"hello-world-g2csp\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""

[cardoe]

I failed to mention that adding --serviceaccount argo to the argo submit made it work.

[terrytangyuan]

Would you like to submit a PR to help improve the docs?

[cardoe]

Would you like to submit a PR to help improve the docs?

Would be glad to. I'm just not sure if that's what the maintainers want from the quick start. If it is then I'll happily update the docs and include a reference to the workflow RBAC for further reading and make a note that when a service account isn't specified then the default account is used which often doesn't have sufficient permissions.

[terrytangyuan]

Looks like the quick start and walk-through guides all use default service account which does not have the permission. We can probably just give default service-account enough permission https://github.com/argoproj/argo-workflows/tree/76f4d1f9ca53b8ac48965e12c15454e544936c49/manifests/quick-start/base

[agilgur5]

Looks like the quick start and walk-through guides all use default service account which does not have the permission.

The default SA actually has a RoleBinding to the executor Role and both are included in the kustomization.yaml so the quick start should work with the default SA already.

kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.2/install.yaml

I think the problem is that the Quick Start docs are referencing install.yaml instead of quick-start-minimal.yaml

[agilgur5]

@cardoe would you like to submit a PR to modify the docs to reference quick-start-minimal.yaml instead of install.yaml?

[cardoe]

Yep will do.

[rnathuji]

I found this issue after stumbling upon the same problem across our workflows when upgrading from v3.5.0 to v3.5.4 using the released manifest files. It seems like this issue was closed after a doc change to the quick start page, but the installation docs and release pages seemingly suggest that one should be able to use install.yaml without issues:

To install Argo Workflows, navigate to the releases page and find the release you wish to use (the latest full release is preferred). Scroll down to the Controller and Server section and execute the kubectl commands.

[agilgur5]

It seems like this issue was closed after a doc change to the quick start page, but the installation docs and release pages seemingly suggest that one should be able to use install.yaml without issue

Yes, this is specific to the quick start, which explicitly states it should not be used in production.

[rnathuji]

Yes, I see that the quick start page links to the install page:

These instructions are intended to help you get started quickly. They are not suitable for production. For production installs, please refer to the installation documentation.

☝️ refers to the release pages which then tells users to run:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/install.yaml

In any case, perhaps it's user / reader error on my part 🤷‍♂️ . Given quick-start-minimal.yaml is not intended for production, and install.yaml doesn't work "out of the box" anymore, is there documentation for what users that were previously using install.yaml should do when updating to v3.5.2 and beyond?

[agilgur5]

and install.yaml doesn't work "out of the box" anymore, is there documentation for what users that were previously using install.yaml should do when updating to v3.5.2 and beyond?

This wasn't listed as a regression, is there a regression in the manifests that you noticed for v3.5.2 vs an older version?
You mention v3.5.0 as your prior, the only changes to the manifests between v3.5.0 and v3.5.2 were #11839 and #11947, and they only changed the full CRDs in the manifests, no RBAC or other resources.

[rnathuji]

Ah, yes, sorry, I wasn't clear about that 😬 . So, for better or worse, we have been using the release doc steps to deploy/update argo via install.yaml since v3.3.9 or so. It's always worked fine without issues. However, when updating from v3.5.0 to 3.5.4 workflows began failing with the message cannot patch resource "pods" in API group "". I did the same diff check between the manifests and the only differences I saw were the image tags:

       - args:
         - server
         env: []
-        image: quay.io/argoproj/argocli:v3.5.0
+        image: quay.io/argoproj/argocli:v3.5.4
         name: argo-server
         ports:
         - containerPort: 2746
@@ -1323,7 +1323,7 @@
             fieldRef:
               apiVersion: v1
               fieldPath: metadata.name
-        image: quay.io/argoproj/workflow-controller:v3.5.0
+        image: quay.io/argoproj/workflow-controller:v3.5.4
         livenessProbe:
           failureThreshold: 3
           httpGet:

That's why we were so surprised to hit the problem. For now we've downgraded to the https://github.com/argoproj/argo-workflows/releases/download/v3.5.1/install.yaml manifest 🤷‍♂️ . Any thoughts / suggestions would be very much appreciated 🙏 .

Update: My comment about 3.5.2 was because while investigating using a local minikube I found that was the first patch version which reproduced the problem.

[agilgur5]

However, when updating from v3.5.0 to 3.5.4 workflows began failing with the message cannot patch resource "pods" in API group ""

So this error happens from the Workflow Pod's Executor, not the Controller. You'll get that error when your Workflow does not have proper RBAC (pod patch is a fallback for when workflowtaskresults RBAC is not provided).

That's why we were so surprised to hit the problem.

So I don't think this is related to the install.yaml, and only tangentially related to this issue. Specifically, the install.yaml is just a concatened cluster install, which hasn't changed outside of the CRDs in ~1.5 years and which contains no RBAC at all for a Workflow / the Executor. That's left entirely to the user to configure. That sounds like it may have been coincidence that it coincided with your upgrade.

[rnathuji]

Thanks, @agilgur5! I will look more into the Workflow RBAC to see what changes might be needed for our workflows to work with v3.5.4.

Just to help clarify my understanding, is the reason why we're seeing existing workflows work when downgrading just the version tag to v3.5.1 in the manifest because we've been historically lucking into some default behavior that no longer applies starting v3.5.2, and the newer versions are simply surfacing a latent RBAC issue on our end? For example, it's why the following works fine:

$ minikube start
$ kubectl create namespace argo
$ kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.1/install.yaml
$ argo submit --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

Output:

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ✔ hello-world-45qtp  whalesay  hello-world-45qtp  18s

But the following will fail (as will versions 3.5.2 and 3.5.3):

$ minikube start
$ kubectl create namespace argo
$ kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/install.yaml
$ argo submit --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

Output:

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-jdsk6  whalesay  hello-world-jdsk6  22s       Error (exit code 1): pods "hello-world-jdsk6" is forbidden: User "system:serviceaccount:default:default" cannot patch resource "pods" in API group "" in the namespace "default"

[agilgur5]

For example, it's why the following works fine:

👀 That's an interesting reproduction.
I thought your prod cluster wasn't fresh, so could be that something else changed in it coincidentally. If this is a fresh minikube cluster each time, then that suggests a change in the Executor is requiring that permission when it previously didn't. Or, more specifically, the simple hello-world example managed to work without any additional RBAC, but now that is required.

I'd need to look through the Executor diffs for 3.5.2 to see why that might happen. From a real quick glance, this might be from #11947 -- these change to wait.go ensure it properly reports results. If that's the case, it may be that some really short, simple steps like hello-world were not recorded entirely correctly previously. Or that they previously skipped some of the reporting that is now required from all steps. Or the context change there might mean that if you were missing RBAC, it might not have finished the request and reported the error before the Pod shut down. There's a few different changes there.
cc @Garett-MacGowan who probably knows this part of the codebase better than me at this point.

[Garett-MacGowan]

I only skimmed this, so im not sure if I'm missing something, but have you configured RBAC according to this?

I can take a better look at this tomorrow.

[agilgur5]

I only skimmed this, so im not sure if I'm missing something, but have you configured RBAC according to this?

I linked that above -- but apparently it wasn't necessary for simple steps like hello-world prior to 3.5.2. #11947 had some Executor changes that seemed like they might've changed some of that logic. It might've been a long-standing bug that didn't have much user-facing impact that you ended up fixing or maybe there's a new request that isn't necessary for simpler steps? I'm not sure, hence why I tagged you.

From my own memory, I wanna say that steps with no inputs or outputs previously worked without workflowtaskresults RBAC; not sure if that was a bug or not though.

I can take a better look at this tomorrow.

Take your time!

[Garett-MacGowan]

@agilgur5 here's my take.

TaskResultReconciliation() which uses workflowtaskresult, for which we need RBAC, is now (necessarily) called before

if woc.wf.Labels[common.LabelKeyCompleted] == "true" { // abort now, we do not want to perform any more processing on a complete workflow because we could corrupt it

Before my PR, if the workflow completed quick enough, reconciliation wouldn't have a chance to run and therefore you would avoid the RBAC issue.

This is closely related to #11947

The quickstart docs should probably mention the need for workflowtaskresult RBAC per https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/

[agilgur5]

Before my PR, if the workflow completed quick enough, reconciliation wouldn't have a chance to run and therefore you would avoid the RBAC issue.

Thought that could be the case per above, thanks for confirming Garett!

TaskResultReconciliation() which uses workflowtaskresult, for which we need RBAC, is now (necessarily) called before

EDIT: wait a minute, isn't that logic in the Controller? The necessary RBAC is for the Workflow/Executor, not the Controller, which has the proper RBAC in any officially supported production install (whereas Workflow RBAC is up to the user to configure). Or am I missing something, like does the Controller reconciliation cause the Executor to respond in some way?

The quickstart docs should probably mention the need for workflowtaskresult RBAC per https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/

The PR that closed this issue, #12445 fixed the quickstart docs to actually use the quick-start-*.yaml at my suggestion above, which has the appropriate permissions.

But the Workflow RBAC page should be linked in the production installation page where it is apparently missing.
@rnathuji would you like to submit a PR to add a link to the Workflow RBAC page to the production installation page? Perhaps it could be made as the first bullet under "Additional installation considerations"

[Garett-MacGowan]

@agilgur5 ahh, yes, you're right. The real reason is here. In this PR, we switched to always creating a task result (a placeholder, if no outputs exist) in order to track reconciliation status.

The implementation has changed slightly since this PR, but it's effectively doing the same thing.

[agilgur5]

Thanks for the context! Thought it might be something like that.

Re-wording, that would mean that previously, as hello-world had no outputs, it would made no task result and hence technically did not need the RBAC. That led to some race conditions however, so now it does create a task result and therefore require the RBAC. The docs have always stated this, so it was actually an undocumented technicality that simpler steps like hello-world didn't need the RBAC in the past.

That also explains why we got a few reports of this on the quick start, since previously the erroneously referenced install.yaml was enough for hello-world, which is also referenced in the quick start. Later pages, in the walk through for instance, did/do require the RBAC though.

[Garett-MacGowan]

Thanks for the context! Thought it might be something like that.

Re-wording, that would mean that previously, as hello-world had no outputs, it would made no task result and hence technically did not need the RBAC. That led to some race conditions however, so now it does create a task result and therefore require the RBAC. The docs have always stated this, so it was actually an undocumented technicality that simpler steps like hello-world didn't need the RBAC in the past.

That's right.

[rnathuji]

@agilgur5 - Sure, I can spin up a doc PR.

Thanks all for taking a look and helping to clarify the reason behind the behavioral change that we observed. 🎉

[agilgur5]

@agilgur5 - Sure, I can spin up a doc PR.

See aforementioned PR in #12680