Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What's causing: forbidden: User "system:anonymous" in some Cloud Providers #225

Closed
klustria opened this Issue Mar 1, 2018 · 20 comments

Comments

Projects
None yet
9 participants
@klustria
Copy link

klustria commented Mar 1, 2018

I deployed API-Server to Google's Managed GKE and Oracle's OCI via Terraform with Unmanaged k8s. The behavior on both these providers are exactly the same. Deploying through Helm would not succeed until adding a ClusterRole like this:

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

Once the API-Server is deployed and running, executing any kubectl command to the Custom Resource would fail:

kubectl get eventprovider
Error from server (Forbidden): eventproviders.auraevents.oracledx.com is forbidden: User "system:anonymous" cannot list eventproviders.auraevents.oracledx.com in the namespace "default"

Any ideas on what would be causing the user to be anonymous leading to this failure?

@pires

This comment has been minimized.

Copy link

pires commented Mar 3, 2018

Most probably you have RBAC enabled, which means that, by default, your cluster won't allow full, unprivileged access any more.

@klustria

This comment has been minimized.

Copy link
Author

klustria commented Mar 3, 2018

Any suggestions on what can be done to help resolve this issue?

@klustria

This comment has been minimized.

Copy link
Author

klustria commented Mar 3, 2018

After reading this:

https://kubernetes.io/docs/admin/authentication/#anonymous-requests

then I tried this:

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

and it solved the problem.

@klustria klustria closed this Mar 3, 2018

@ghuntley

This comment has been minimized.

Copy link

ghuntley commented Jun 7, 2018

@klustria that command you have executed uplifts all anonymous requests to super-admin level access. If this instance is publically running your probs hacked by now and data has been exfiltrated. :/

@schittler

This comment has been minimized.

Copy link
Contributor

schittler commented Jun 7, 2018

Well, sort of @ghuntley

This is concerning RBAC, not access to kubectl. The externally reachable apiserver endpoint has some other authentication mechanism (usually TLS client certificates, but things like OIDC are possible too). RBAC is an additional layer of security that many clusters don't even use. The command above is just a novel way to set --authorization-mode=AllowAll

Anyway, back to the issue. We faced this problem too. When RBAC is enabled, the aggregated apiserver somehow has to figure out what user is sending a request. It then asks the kubernetes-apiserver if that user is allowed to perform said action. If the mechanism to tell the aggregated server the user is not working, it defaults to user:anonymous, which is then rejected.

You need to add some flags to kubernetes-apiserver for this to work, check here

@klustria

This comment has been minimized.

Copy link
Author

klustria commented Jun 7, 2018

Appreciate the discussion still ongoing around this topic. I know that above approach I did is just a workaround and since we are not in production, I'm using it to get us going. @damongant, what exact flags from https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/ did you use exactly to make this work for you?

@schittler

This comment has been minimized.

Copy link
Contributor

schittler commented Jun 7, 2018

apiserver:

      --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem
      --requestheader-allowed-names=aggregator
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
@klustria

This comment has been minimized.

Copy link
Author

klustria commented Jun 7, 2018

And this needs to be set on the Core APIServer side, not the aggregated APIServer?

@schittler

This comment has been minimized.

Copy link
Contributor

schittler commented Jun 7, 2018

Yes

@klustria

This comment has been minimized.

Copy link
Author

klustria commented Jun 7, 2018

Thanks I will try it out. The challenge of course is if you are working with Managed Kubernetes Clusters where you would have little to no ability to change the flags for the APIServer. If I recall, GKE has this issue too and I'm not sure if these flags can be set there. Currently, I'm using Unmanaged Cluster via terraform so I can try out those flags to see if it will work for me.

@markusdresch

This comment has been minimized.

Copy link

markusdresch commented Jun 11, 2018

i managed to create a workaround by applying this yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: anonymous-logs-role
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["create", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: anonymous-logs-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: anonymous-logs-role
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous

this way logs, exec and attach work and the api server doesn't expose any information, so this isn't as permissive as AllowAll. any drawbacks?

@schittler

This comment has been minimized.

Copy link
Contributor

schittler commented Jun 11, 2018

You can't apply RBAC rules to the aggregated types if you do this. The aggregated server is always going to see the connecting user as system:anonymous.

@klustria

This comment has been minimized.

Copy link
Author

klustria commented Jun 19, 2018

Hi @damongant, I added:

  --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem
  --requestheader-allowed-names=aggregator
  --requestheader-extra-headers-prefix=X-Remote-Extra-
  --requestheader-group-headers=X-Remote-Group
  --requestheader-username-headers=X-Remote-User

into the K8s Masters Core APIServer and I'm still getting the same issue:

Error from server (Forbidden): error when creating "config/aieventprovider.yaml": eventproviders.auraevents.oracledx.com is forbidden: User "system:anonymous" cannot create eventproviders.auraevents.oracledx.com in the namespace "default"

Are there any other settings I need to set like in the extension API Server or anywhere else?

@rbjorklin

This comment has been minimized.

Copy link

rbjorklin commented Aug 16, 2018

@klustria I also had to add:

--proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>

as described here. Make sure the proxy certificate maps to a user with the correct permissions/rolebindings.

@schittler

This comment has been minimized.

Copy link
Contributor

schittler commented Oct 22, 2018

The core apiserver still knows what user you are, but as soon as its proxied to the aggregated server it doesn't know anymore (because headers and TLS credentials don't "punch through") and the flags described above need to be set up for the aggregated server to not assume every request is anonymous.

@wuqifu

This comment has been minimized.

Copy link

wuqifu commented Dec 5, 2018

After reading this:

https://kubernetes.io/docs/admin/authentication/#anonymous-requests

then I tried this:

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

and it solved the problem.

works for me.

@yue9944882

This comment has been minimized.

Copy link
Member

yue9944882 commented Dec 5, 2018

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

this is not recommended since that it will grant extra privileges to any requests w/ anonymous access.
apiserver will be authenticating a request as "anonymous" only if you enabled --anonymous-auth(which is enabled by default) but didn't configure other authenticator like request proxy headers. this error message might be suggesting that you should configure at least one authenticator for ur aggregated server.

@yue9944882

This comment has been minimized.

Copy link
Member

yue9944882 commented Dec 5, 2018

except for request header authenticator, you could also use delegated auth (via Authn/AuthzReview API)

note that things that use delegated authentication should have:

  • the extension-apiserver-authentication-reader role in the kube-system namespace
  • the system:auth-delegator clusterrole at the cluster scope

FYI https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md#rbac-rules

@yue9944882

This comment has been minimized.

Copy link
Member

yue9944882 commented Dec 6, 2018

enabling anonymous auth is evil for aggregated server, linking recent CVE:

kubernetes/kubernetes#71411

will disable that by default in this builder

@jsloyer

This comment has been minimized.

Copy link

jsloyer commented Mar 15, 2019

Do not do what #225 (comment) is suggesting. Huge security hole....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.