EKS authentication requires newer client-go

Getting the following error when trying to use kubeseal with EKS.

$ kubeseal --fetch-cert
panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

I am able to interact with the cluster normally using kubectl.

$ kubectl get pods -n kube-system
NAME                                         READY     STATUS    RESTARTS   AGE
aws-node-68vfm                               1/1       Running   0          6d
aws-node-hl9ww                               1/1       Running   1          6d
aws-node-nhkwb                               1/1       Running   1          6d
fluentd-78l2j                                1/1       Running   0          6d
fluentd-kfdn4                                1/1       Running   0          6d
fluentd-wfj48                                1/1       Running   0          6d
kube-dns-7cc87d595-fbmxq                     3/3       Running   0          6d
kube-proxy-fkpsw                             1/1       Running   0          6d
kube-proxy-lbwvz                             1/1       Running   0          6d
kube-proxy-qqs2p                             1/1       Running   0          6d
sealed-secrets-controller-5bc5488bb4-79blg   1/1       Running   0          22m

[anguslees]

Interesting, thanks for the report.

As background explanation: --fetch-cert requires reading the public-key portion of the sealed-secrets master Secret in kube-system somehow. Accessing the Secret directly works, but is a bit scary security-wise (since the private-key is there too), so instead I serve just the public-key portion via HTTP from the sealed-secrets controller, and then kubeseal fetches it via a proxy through the k8s apiserver. This is why it requires services/proxy in kube-system.

So, your options are:

• Retry with a k8s user with greater access. "system:anonymous" sounds like you are using an unauthenticated user of some sort.
• If you can read the logs from the sealed-secrets-controller pod (kubectl -n kube-system logs sealed-secrets-controller-5bc5488bb4-79blg in your example above), then the public key is written to the log on startup(*). I expect system:anonymous will not grant sufficient access for this either.
• Get to the http://sealed-secrets-controller.kube-system:8080/v1/cert.pem in-cluster URL some other way, perhaps from a pod running in the cluster.

(*) The public key (via any method) is the text between and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

It doesn't matter which user you use to get the public key, it's the same public key for all users of that cluster. So typically the workflow would be: admin installs sealed-secret controller, admin runs kubeseal --fetch-cert, admin publishes cert via some other means (intranet webpage, git repo, etc), regular users use cert.

For this bug:

It would be nice if we could make this easier for future users of EKS. I haven't used EKS - how did you get these credentials to the EKS cluster? Is it going to be typical for EKS users to use system:anonymous in this way? If so, we might need to explicitly add a more liberal services/proxy RBAC rule for the controller http service (it is meant to be ~widely accessible).

My kubeconfig is setup using eksctl:

eksctl utils write-kubeconfig my-cluster --profile my-profile

Relevant code is here: https://github.com/weaveworks/eksctl/blob/master/cmd/eksctl/utils.go#L85-L164

EKS requires aws-iam-authenticator to auth with k8s. It was formerly called heptio-authenticator-aws and is vendored in eksutil here: https://github.com/weaveworks/eksctl/tree/master/vendor/github.com/heptio/authenticator/cmd/heptio-authenticator-aws.

I'm able to view logs for the controller pod and get the certificate that way. I'll try updating RBAC tomorrow.

My temporary solution:

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')

[anguslees]

Huh, I think I originally misunderstood the issue here. This isn't about RBAC, since we should not allow system:anonymous to access the service (there's nothing actually wrong with this since the pubkey is supposed to be public, but system:anonymous represents API requests that have not provided any authentication token and probably should not have any access to the cluster at all). As @jramos last comment shows, if we provide a valid token, then we're no longer system:anonymous and everything works.

So the issue here is that the API request that kubeseal makes does not include the AWS bearer token. The fix appears to require updating to a version of client-go that includes the "exec" auth plugin, which is alpha in 1.10 / beta in k8s 1.11.

Meanwhile, the correct workaround is the explicit --token arg from the previous comment.

👍 thanks for the explanation.

[stromvirvel]

just for information:

This issue obviously also pops up when you're trying to seal a secret:

kubeseal  < secrets.yaml
Please enter Username: Please enter Password: **panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

I wasted a lot of time to find out what the problem is here, until I found this issue. Maybe you could mention the workaround in the README?

[admssa]

Hi guys, do you have any ETA for updating client-go lib and rebuild?
Or maybe you have some beta binaries?
And one more thing - can I set controller IP manually? It tries to connect directly to pod.

admssa$ kubeseal --fetch-cert --token $(aws-iam-authenticator token -i cluster-name  | jq -r '.status.token')

panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.237.72.52:8080: getsockopt: connection timed out'\nTrying to reach: 'http://10.237.72.52:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

goroutine 1 [running]:

[anguslees]

can I set controller IP manually?

No. As currently written, kubeseal talks to the controller via the apiserver "proxy" to a known service (by namespace+name).

You only need this for fetching the public key though - you can run kubeseal --fetch-cert once, and then pass the certificate file to later invocations of kubeseal --cert. If you do this, then kubeseal doesn't talk to the controller at all during sealing. This is the recommended approach for automated pipelines (since it removes the need to give your robots authenticated k8s API access).