po/tf-job-operator logs said user cannot get endpoints in the namespace "default"

[wydwww]

After ks apply default -c kubeflow-core, the jupyterhub was up but when I checked kubectl logs tf-job-operator-77b689c87d-g7w2z, it said

E0110 11:10:03.455096       1 election.go:226] error retrieving resource lock default/tf-operator: endpoints "tf-operator" is forbidden: User "system:serviceaccount:default:tf-job-operator" cannot get endpoints in the namespace "default"

It's pretty same with kubeflow/training-operator#173 but it's not deployed by helm, so how to fix this?

Thanks

[foxish]

This is an RBAC issue - you'd need to create a rolebinding that grants the ability to view the endpoints resource to default:tf-job-operator. https://kubernetes.io/docs/admin/authorization/rbac/ is a good starting point on RBAC.

[wydwww]

@foxish Thanks! It's easy to fix

kubectl create clusterrolebinding tf-job-rule --clusterrole=cluster-admin --serviceaccount=default:tf-job-operator

Do I need to add this to https://github.com/google/kubeflow#rbac-clusters?

[jlewi]

The ksonnet configs should create the necessary roles. So if that's not happening its a bug and we should figure out why and fix it.