Insufficient description in the doc regarding RBAC configuration

[huikang]

I am following the get-started tutorial to install the argo workflow 2.7.
The only change I made to the install yaml is to add containerRuntimeExecutor: kubelet in the ConfigMap.

Then I run the following command following the instruction:

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

However, after I submit a job, the following error appears

argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml

failed to save outputs: unexpected non 200 status code: 403, body: Forbidden (user=system:serviceaccount:default:default, verb=get, resource=nodes, subresource=proxy)

Could anyone help to look at this issue? Thanks.

[huikang]

My understanding is that I need to assign proper permission to access the resource nodes and nodes/proxy. Could anyone offer some help about how to setup this for argo? Thanks.

[simster7]

Correct, you need to add permissions to access them by binding them to a service account(either default:default or another one you create). You can use this as an example on how to do so: https://github.com/argoproj/argo/blob/master/docs/workflow-rbac.md

Should add a new resource nodes and add the proper verbs.

Hint: I use this a lot when checking if I was successful in binding new permissions correctly: https://stackoverflow.com/questions/54889458/kubernetes-check-serviceaccount-permissions

[huikang]

Hi, @simster7 , I applied the following role yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-role
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
  - ""
  resources:
  - pods
  - nodes/proxy
  verbs:
  - get
  - watch
  - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get
  - watch

However, I am still getting the same error and seems not having the permission to get nodes/proxy

kubectl auth can-i get nodes/proxy --as=system:serviceaccount:default:default   -n default      
Warning: resource 'nodes' is not namespace scoped
no

Did I miss any step? Thanks.

[huikang]

Even I did a role binding as followings

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-role
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  verbs:
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-workflow-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

kubectl auth can-i get nodes/proxy --as=system:serviceaccount:default:default   -n default
Warning: resource 'nodes' is not namespace scoped
yes

The submited job still returns an error

argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml

failed to save outputs: unexpected non 200 status code: 403, body: Forbidden (user=system:serviceaccount:default:default, verb=get, resource=nodes, subresource=proxy)

[lippertmarkus]

Resource nodes/proxy is non-namespaced (a cluster-wide resource) so instead of a Role and a RoleBinding you need to create a ClusterRole and a ClusterRoleBinding:

For testing you can use

kubectl create clusterrolebinding default-admin --clusterrole=admin --serviceaccount=default:default

but it should also work when you change the kind of your manifests above to ClusterRole / ClusterRoleBinding accordingly.

This is only necessary when using kubelet executor and should be updated in the docs.

[h4wkmoon]

Hi,

For testing you can use

And for non-testing purpose, what should we do ?

Edit: I realised I haven't been very polite. Sorry.
Job scheduling is something we need on kubernetes, and openshift. Thanks.

[lippertmarkus]

@h4wkmoon I use the following ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-executor
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
  - ""
  resources:
  - pods/log
  - nodes/proxy
  verbs:
  - get
  - watch

[gasgithub]

This doesnt work even after defining:

kubectl create clusterrolebinding default-admin --clusterrole=admin --serviceaccount=argo:default

I'm still getting:

oc auth can-i get nodes/proxy --as=system:serviceaccount:argo:default
Warning: resource 'nodes' is not namespace scoped
no

and error like:

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-qfcvk  whalesay  hello-world-qfcvk  7s        failed to save outputs: unexpected non 200 status code: 403, body: Forbidden (user=system:serviceaccount:argo:default, verb=get, resource=nodes, subresource=proxy)

Anything missing?? Could you please give any hints.

Any way to force step pods to run on non default account? As I'm able to configure argo service account to have correct permissions but not the default...

oc auth can-i get nodes/proxy --as=system:serviceaccount:argo:argo
Warning: resource 'nodes' is not namespace scoped
yes

Well, although I changed workflow.spec.serviceAccountName=argo, it didnt help as the error still appears:

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-m45vg  whalesay  hello-world-m45vg  8s        failed to save outputs: unexpected non 200 status code: 403, body: Forbidden (user=system:serviceaccount:argo:argo, verb=get, resource=nodes, subresource=proxy)

So unfortunately there is still something missing...