New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EKS authentication requires newer client-go #110

Open
jramos opened this Issue Aug 15, 2018 · 12 comments

Comments

Projects
None yet
7 participants
@jramos
Copy link

jramos commented Aug 15, 2018

Getting the following error when trying to use kubeseal with EKS.

$ kubeseal --fetch-cert
panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

I am able to interact with the cluster normally using kubectl.

$ kubectl get pods -n kube-system
NAME                                         READY     STATUS    RESTARTS   AGE
aws-node-68vfm                               1/1       Running   0          6d
aws-node-hl9ww                               1/1       Running   1          6d
aws-node-nhkwb                               1/1       Running   1          6d
fluentd-78l2j                                1/1       Running   0          6d
fluentd-kfdn4                                1/1       Running   0          6d
fluentd-wfj48                                1/1       Running   0          6d
kube-dns-7cc87d595-fbmxq                     3/3       Running   0          6d
kube-proxy-fkpsw                             1/1       Running   0          6d
kube-proxy-lbwvz                             1/1       Running   0          6d
kube-proxy-qqs2p                             1/1       Running   0          6d
sealed-secrets-controller-5bc5488bb4-79blg   1/1       Running   0          22m
@anguslees

This comment has been minimized.

Copy link
Contributor

anguslees commented Aug 16, 2018

Interesting, thanks for the report.

As background explanation: --fetch-cert requires reading the public-key portion of the sealed-secrets master Secret in kube-system somehow. Accessing the Secret directly works, but is a bit scary security-wise (since the private-key is there too), so instead I serve just the public-key portion via HTTP from the sealed-secrets controller, and then kubeseal fetches it via a proxy through the k8s apiserver. This is why it requires services/proxy in kube-system.

So, your options are:

  • Retry with a k8s user with greater access. "system:anonymous" sounds like you are using an unauthenticated user of some sort.
  • If you can read the logs from the sealed-secrets-controller pod (kubectl -n kube-system logs sealed-secrets-controller-5bc5488bb4-79blg in your example above), then the public key is written to the log on startup(*). I expect system:anonymous will not grant sufficient access for this either.
  • Get to the http://sealed-secrets-controller.kube-system:8080/v1/cert.pem in-cluster URL some other way, perhaps from a pod running in the cluster.

(*) The public key (via any method) is the text between and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

It doesn't matter which user you use to get the public key, it's the same public key for all users of that cluster. So typically the workflow would be: admin installs sealed-secret controller, admin runs kubeseal --fetch-cert, admin publishes cert via some other means (intranet webpage, git repo, etc), regular users use cert.

For this bug:

It would be nice if we could make this easier for future users of EKS. I haven't used EKS - how did you get these credentials to the EKS cluster? Is it going to be typical for EKS users to use system:anonymous in this way? If so, we might need to explicitly add a more liberal services/proxy RBAC rule for the controller http service (it is meant to be ~widely accessible).

@jramos

This comment has been minimized.

Copy link
Author

jramos commented Aug 16, 2018

My kubeconfig is setup using eksctl:

eksctl utils write-kubeconfig my-cluster --profile my-profile

Relevant code is here: https://github.com/weaveworks/eksctl/blob/master/cmd/eksctl/utils.go#L85-L164

EKS requires aws-iam-authenticator to auth with k8s. It was formerly called heptio-authenticator-aws and is vendored in eksutil here: https://github.com/weaveworks/eksctl/tree/master/vendor/github.com/heptio/authenticator/cmd/heptio-authenticator-aws.

@jramos

This comment has been minimized.

Copy link
Author

jramos commented Aug 16, 2018

I'm able to view logs for the controller pod and get the certificate that way. I'll try updating RBAC tomorrow.

@jramos

This comment has been minimized.

Copy link
Author

jramos commented Aug 16, 2018

My temporary solution:

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
@anguslees

This comment has been minimized.

Copy link
Contributor

anguslees commented Aug 21, 2018

Huh, I think I originally misunderstood the issue here. This isn't about RBAC, since we should not allow system:anonymous to access the service (there's nothing actually wrong with this since the pubkey is supposed to be public, but system:anonymous represents API requests that have not provided any authentication token and probably should not have any access to the cluster at all). As @jramos last comment shows, if we provide a valid token, then we're no longer system:anonymous and everything works.

So the issue here is that the API request that kubeseal makes does not include the AWS bearer token. The fix appears to require updating to a version of client-go that includes the "exec" auth plugin, which is alpha in 1.10 / beta in k8s 1.11.

Meanwhile, the correct workaround is the explicit --token arg from the previous comment.

@anguslees anguslees changed the title panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system" EKS authentication requires newer client-go Aug 21, 2018

@jramos

This comment has been minimized.

Copy link
Author

jramos commented Aug 21, 2018

👍 thanks for the explanation.

@moepot

This comment has been minimized.

Copy link

moepot commented Oct 11, 2018

just for information:

This issue obviously also pops up when you're trying to seal a secret:

kubeseal  < secrets.yaml
Please enter Username: Please enter Password: **panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

I wasted a lot of time to find out what the problem is here, until I found this issue. Maybe you could mention the workaround in the README?

@admssa

This comment has been minimized.

Copy link

admssa commented Nov 7, 2018

Hi guys, do you have any ETA for updating client-go lib and rebuild?
Or maybe you have some beta binaries?
And one more thing - can I set controller IP manually? It tries to connect directly to pod.

admssa$ kubeseal --fetch-cert --token $(aws-iam-authenticator token -i cluster-name  | jq -r '.status.token')

panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.237.72.52:8080: getsockopt: connection timed out'\nTrying to reach: 'http://10.237.72.52:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

goroutine 1 [running]:

jipperinbham added a commit to jipperinbham/sealed-secrets that referenced this issue Nov 9, 2018

client-go version bump to release-7.0
also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses bitnami-labs#110
@anguslees

This comment has been minimized.

Copy link
Contributor

anguslees commented Nov 13, 2018

can I set controller IP manually?

No. As currently written, kubeseal talks to the controller via the apiserver "proxy" to a known service (by namespace+name).

You only need this for fetching the public key though - you can run kubeseal --fetch-cert once, and then pass the certificate file to later invocations of kubeseal --cert. If you do this, then kubeseal doesn't talk to the controller at all during sealing. This is the recommended approach for automated pipelines (since it removes the need to give your robots authenticated k8s API access).

jipperinbham added a commit to jipperinbham/sealed-secrets that referenced this issue Nov 13, 2018

client-go version bump to release-7.0
also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses bitnami-labs#110

bors bot added a commit that referenced this issue Nov 13, 2018

Merge #126
126: client-go version bump to release-7.0 r=anguslees a=jipperinbham

also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses #110 

Co-authored-by: JP Phillips <jonphill9@gmail.com>
@JessieAMorris

This comment has been minimized.

Copy link

JessieAMorris commented Dec 19, 2018

It looks like this has been resolved with #126. I just tested on an EKS cluster and it is working correctly for me.

@NeoTech

This comment has been minimized.

Copy link

NeoTech commented Jan 25, 2019

I found this and tried the "hack" i keep crashing tho for an AWS EKS cluster.

> kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
@morganchristiansson

This comment has been minimized.

Copy link

morganchristiansson commented Feb 18, 2019

I'm getting same error as @NeoTech

EDIT: Managed workaround. Use local secret: kubeseal --cert sealed-secrets-key.yaml --fetch-cert

EDIT2: Still not working. output is supposed to be a pem file but I'm getting yaml output. But my workaround to use entirely local workflow is valid just --fetch-cert doesn't work correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment