Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What's causing: forbidden: User "system:anonymous" in some Cloud Providers #225

Closed
klustria opened this issue Mar 1, 2018 · 21 comments
Closed

Comments

@klustria
Copy link

@klustria klustria commented Mar 1, 2018

I deployed API-Server to Google's Managed GKE and Oracle's OCI via Terraform with Unmanaged k8s. The behavior on both these providers are exactly the same. Deploying through Helm would not succeed until adding a ClusterRole like this:

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

Once the API-Server is deployed and running, executing any kubectl command to the Custom Resource would fail:

kubectl get eventprovider
Error from server (Forbidden): eventproviders.auraevents.oracledx.com is forbidden: User "system:anonymous" cannot list eventproviders.auraevents.oracledx.com in the namespace "default"

Any ideas on what would be causing the user to be anonymous leading to this failure?

@pires
Copy link

@pires pires commented Mar 3, 2018

Most probably you have RBAC enabled, which means that, by default, your cluster won't allow full, unprivileged access any more.

@klustria
Copy link
Author

@klustria klustria commented Mar 3, 2018

Any suggestions on what can be done to help resolve this issue?

@klustria
Copy link
Author

@klustria klustria commented Mar 3, 2018

After reading this:

https-:-//kubernetes.io/docs/admin/authentication/#anonymous-requests

then I tried this:

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=dont-do-this --user=system:anonymous

and it solved the problem.

edit from Kubernetes team:
Please see the official response from the Kubernetes project:
#225 (comment)

@klustria klustria closed this Mar 3, 2018
@ghuntley
Copy link

@ghuntley ghuntley commented Jun 7, 2018

@klustria that command you have executed uplifts all anonymous requests to super-admin level access. If this instance is publically running your probs hacked by now and data has been exfiltrated. :/

@acuteaura
Copy link
Contributor

@acuteaura acuteaura commented Jun 7, 2018

Well, sort of @ghuntley

This is concerning RBAC, not access to kubectl. The externally reachable apiserver endpoint has some other authentication mechanism (usually TLS client certificates, but things like OIDC are possible too). RBAC is an additional layer of security that many clusters don't even use. The command above is just a novel way to set --authorization-mode=AllowAll

Anyway, back to the issue. We faced this problem too. When RBAC is enabled, the aggregated apiserver somehow has to figure out what user is sending a request. It then asks the kubernetes-apiserver if that user is allowed to perform said action. If the mechanism to tell the aggregated server the user is not working, it defaults to user:anonymous, which is then rejected.

You need to add some flags to kubernetes-apiserver for this to work, check here

@klustria
Copy link
Author

@klustria klustria commented Jun 7, 2018

Appreciate the discussion still ongoing around this topic. I know that above approach I did is just a workaround and since we are not in production, I'm using it to get us going. @damongant, what exact flags from https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/ did you use exactly to make this work for you?

@acuteaura
Copy link
Contributor

@acuteaura acuteaura commented Jun 7, 2018

apiserver:

      --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem
      --requestheader-allowed-names=aggregator
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
@klustria
Copy link
Author

@klustria klustria commented Jun 7, 2018

And this needs to be set on the Core APIServer side, not the aggregated APIServer?

@acuteaura
Copy link
Contributor

@acuteaura acuteaura commented Jun 7, 2018

Yes

@klustria
Copy link
Author

@klustria klustria commented Jun 7, 2018

Thanks I will try it out. The challenge of course is if you are working with Managed Kubernetes Clusters where you would have little to no ability to change the flags for the APIServer. If I recall, GKE has this issue too and I'm not sure if these flags can be set there. Currently, I'm using Unmanaged Cluster via terraform so I can try out those flags to see if it will work for me.

@markusdresch
Copy link

@markusdresch markusdresch commented Jun 11, 2018

i managed to create a workaround by applying this yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: anonymous-logs-role
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["create", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: anonymous-logs-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: anonymous-logs-role
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous

this way logs, exec and attach work and the api server doesn't expose any information, so this isn't as permissive as AllowAll. any drawbacks?

@acuteaura
Copy link
Contributor

@acuteaura acuteaura commented Jun 11, 2018

You can't apply RBAC rules to the aggregated types if you do this. The aggregated server is always going to see the connecting user as system:anonymous.

@klustria
Copy link
Author

@klustria klustria commented Jun 19, 2018

Hi @damongant, I added:

  --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem
  --requestheader-allowed-names=aggregator
  --requestheader-extra-headers-prefix=X-Remote-Extra-
  --requestheader-group-headers=X-Remote-Group
  --requestheader-username-headers=X-Remote-User

into the K8s Masters Core APIServer and I'm still getting the same issue:

Error from server (Forbidden): error when creating "config/aieventprovider.yaml": eventproviders.auraevents.oracledx.com is forbidden: User "system:anonymous" cannot create eventproviders.auraevents.oracledx.com in the namespace "default"

Are there any other settings I need to set like in the extension API Server or anywhere else?

@rbjorklin
Copy link

@rbjorklin rbjorklin commented Aug 16, 2018

@klustria I also had to add:

--proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>

as described here. Make sure the proxy certificate maps to a user with the correct permissions/rolebindings.

@acuteaura
Copy link
Contributor

@acuteaura acuteaura commented Oct 22, 2018

The core apiserver still knows what user you are, but as soon as its proxied to the aggregated server it doesn't know anymore (because headers and TLS credentials don't "punch through") and the flags described above need to be set up for the aggregated server to not assume every request is anonymous.

@wuqifu
Copy link

@wuqifu wuqifu commented Dec 5, 2018

works for me.

edit from Kubernetes team:
Please see the official response from the Kubernetes project:
#225 (comment)

@yue9944882
Copy link
Member

@yue9944882 yue9944882 commented Dec 5, 2018

this is not recommended since that it will grant extra privileges to any requests w/ anonymous access.
apiserver will be authenticating a request as "anonymous" only if you enabled --anonymous-auth(which is enabled by default) but didn't configure other authenticator like request proxy headers. this error message might be suggesting that you should configure at least one authenticator for ur aggregated server.

edit from Kubernetes team:
Please see the official response from the Kubernetes project:
#225 (comment)

@yue9944882
Copy link
Member

@yue9944882 yue9944882 commented Dec 5, 2018

except for request header authenticator, you could also use delegated auth (via Authn/AuthzReview API)

note that things that use delegated authentication should have:

  • the extension-apiserver-authentication-reader role in the kube-system namespace
  • the system:auth-delegator clusterrole at the cluster scope

FYI https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md#rbac-rules

@yue9944882
Copy link
Member

@yue9944882 yue9944882 commented Dec 6, 2018

enabling anonymous auth is evil for aggregated server, linking recent CVE:

kubernetes/kubernetes#71411

will disable that by default in this builder

@jsloyer
Copy link

@jsloyer jsloyer commented Mar 15, 2019

Do not do what #225 (comment) is suggesting. Huge security hole....

@mikedanese
Copy link

@mikedanese mikedanese commented Jun 12, 2019

The Kubernetes team considers some of the suggestions made in this thread to be harmful. Some suggestions recommend granting anonymous clients full access to the Kubernetes API and should not be considered as solutions to permission issues. While anonymous auth is useful in specific circumstances, it should only be applied with care and understanding of the Kubernetes API and permission model.

For more information on RBAC, please check out the following resources:

https://kubernetes.io/docs/reference/access-authn-authz/rbac/
https://www.cncf.io/blog/2018/08/01/demystifying-rbac-in-kubernetes/

@kubernetes-sigs kubernetes-sigs locked and limited conversation to collaborators Jun 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants