Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EKS authentication requires newer client-go #110

Closed
ghost opened this issue Aug 15, 2018 · 26 comments
Closed

EKS authentication requires newer client-go #110

ghost opened this issue Aug 15, 2018 · 26 comments
Milestone

Comments

@ghost
Copy link

@ghost ghost commented Aug 15, 2018

Getting the following error when trying to use kubeseal with EKS.

$ kubeseal --fetch-cert
panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

I am able to interact with the cluster normally using kubectl.

$ kubectl get pods -n kube-system
NAME                                         READY     STATUS    RESTARTS   AGE
aws-node-68vfm                               1/1       Running   0          6d
aws-node-hl9ww                               1/1       Running   1          6d
aws-node-nhkwb                               1/1       Running   1          6d
fluentd-78l2j                                1/1       Running   0          6d
fluentd-kfdn4                                1/1       Running   0          6d
fluentd-wfj48                                1/1       Running   0          6d
kube-dns-7cc87d595-fbmxq                     3/3       Running   0          6d
kube-proxy-fkpsw                             1/1       Running   0          6d
kube-proxy-lbwvz                             1/1       Running   0          6d
kube-proxy-qqs2p                             1/1       Running   0          6d
sealed-secrets-controller-5bc5488bb4-79blg   1/1       Running   0          22m
@anguslees
Copy link
Contributor

@anguslees anguslees commented Aug 16, 2018

Interesting, thanks for the report.

As background explanation: --fetch-cert requires reading the public-key portion of the sealed-secrets master Secret in kube-system somehow. Accessing the Secret directly works, but is a bit scary security-wise (since the private-key is there too), so instead I serve just the public-key portion via HTTP from the sealed-secrets controller, and then kubeseal fetches it via a proxy through the k8s apiserver. This is why it requires services/proxy in kube-system.

So, your options are:

  • Retry with a k8s user with greater access. "system:anonymous" sounds like you are using an unauthenticated user of some sort.
  • If you can read the logs from the sealed-secrets-controller pod (kubectl -n kube-system logs sealed-secrets-controller-5bc5488bb4-79blg in your example above), then the public key is written to the log on startup(*). I expect system:anonymous will not grant sufficient access for this either.
  • Get to the http://sealed-secrets-controller.kube-system:8080/v1/cert.pem in-cluster URL some other way, perhaps from a pod running in the cluster.

(*) The public key (via any method) is the text between and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

It doesn't matter which user you use to get the public key, it's the same public key for all users of that cluster. So typically the workflow would be: admin installs sealed-secret controller, admin runs kubeseal --fetch-cert, admin publishes cert via some other means (intranet webpage, git repo, etc), regular users use cert.

For this bug:

It would be nice if we could make this easier for future users of EKS. I haven't used EKS - how did you get these credentials to the EKS cluster? Is it going to be typical for EKS users to use system:anonymous in this way? If so, we might need to explicitly add a more liberal services/proxy RBAC rule for the controller http service (it is meant to be ~widely accessible).

@ghost
Copy link
Author

@ghost ghost commented Aug 16, 2018

My kubeconfig is setup using eksctl:

eksctl utils write-kubeconfig my-cluster --profile my-profile

Relevant code is here: https://github.com/weaveworks/eksctl/blob/master/cmd/eksctl/utils.go#L85-L164

EKS requires aws-iam-authenticator to auth with k8s. It was formerly called heptio-authenticator-aws and is vendored in eksutil here: https://github.com/weaveworks/eksctl/tree/master/vendor/github.com/heptio/authenticator/cmd/heptio-authenticator-aws.

@ghost
Copy link
Author

@ghost ghost commented Aug 16, 2018

I'm able to view logs for the controller pod and get the certificate that way. I'll try updating RBAC tomorrow.

@ghost
Copy link
Author

@ghost ghost commented Aug 16, 2018

My temporary solution:

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
@anguslees
Copy link
Contributor

@anguslees anguslees commented Aug 21, 2018

Huh, I think I originally misunderstood the issue here. This isn't about RBAC, since we should not allow system:anonymous to access the service (there's nothing actually wrong with this since the pubkey is supposed to be public, but system:anonymous represents API requests that have not provided any authentication token and probably should not have any access to the cluster at all). As @jramos last comment shows, if we provide a valid token, then we're no longer system:anonymous and everything works.

So the issue here is that the API request that kubeseal makes does not include the AWS bearer token. The fix appears to require updating to a version of client-go that includes the "exec" auth plugin, which is alpha in 1.10 / beta in k8s 1.11.

Meanwhile, the correct workaround is the explicit --token arg from the previous comment.

@anguslees anguslees changed the title panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system" EKS authentication requires newer client-go Aug 21, 2018
@ghost
Copy link
Author

@ghost ghost commented Aug 21, 2018

👍 thanks for the explanation.

@stromvirvel
Copy link

@stromvirvel stromvirvel commented Oct 11, 2018

just for information:

This issue obviously also pops up when you're trying to seal a secret:

kubeseal  < secrets.yaml
Please enter Username: Please enter Password: **panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

I wasted a lot of time to find out what the problem is here, until I found this issue. Maybe you could mention the workaround in the README?

@admssa
Copy link

@admssa admssa commented Nov 7, 2018

Hi guys, do you have any ETA for updating client-go lib and rebuild?
Or maybe you have some beta binaries?
And one more thing - can I set controller IP manually? It tries to connect directly to pod.

admssa$ kubeseal --fetch-cert --token $(aws-iam-authenticator token -i cluster-name  | jq -r '.status.token')

panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.237.72.52:8080: getsockopt: connection timed out'\nTrying to reach: 'http://10.237.72.52:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

goroutine 1 [running]:
jipperinbham added a commit to jipperinbham/sealed-secrets that referenced this issue Nov 9, 2018
also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses bitnami-labs#110
@anguslees
Copy link
Contributor

@anguslees anguslees commented Nov 13, 2018

can I set controller IP manually?

No. As currently written, kubeseal talks to the controller via the apiserver "proxy" to a known service (by namespace+name).

You only need this for fetching the public key though - you can run kubeseal --fetch-cert once, and then pass the certificate file to later invocations of kubeseal --cert. If you do this, then kubeseal doesn't talk to the controller at all during sealing. This is the recommended approach for automated pipelines (since it removes the need to give your robots authenticated k8s API access).

jipperinbham added a commit to jipperinbham/sealed-secrets that referenced this issue Nov 13, 2018
also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses bitnami-labs#110
bors bot added a commit that referenced this issue Nov 13, 2018
Merge #126
126: client-go version bump to release-7.0 r=anguslees a=jipperinbham

also includes version bumps of k8s.io/api and k8s.io/apimachinery to release-1.10 with a new vendor of github.com/json-iterator/go due to an issue (kubernetes/apimachinery#46) with client-go v7.0

addresses #110 

Co-authored-by: JP Phillips <jonphill9@gmail.com>
@JessieAMorris
Copy link

@JessieAMorris JessieAMorris commented Dec 19, 2018

It looks like this has been resolved with #126. I just tested on an EKS cluster and it is working correctly for me.

@NeoTech
Copy link

@NeoTech NeoTech commented Jan 25, 2019

I found this and tried the "hack" i keep crashing tho for an AWS EKS cluster.

> kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
@morganchristiansson
Copy link

@morganchristiansson morganchristiansson commented Feb 18, 2019

I'm getting same error as @NeoTech

EDIT: Managed workaround. Use local secret: kubeseal --cert sealed-secrets-key.yaml --fetch-cert

EDIT2: Still not working. output is supposed to be a pem file but I'm getting yaml output. But my workaround to use entirely local workflow is valid just --fetch-cert doesn't work correctly

@damascenorakuten
Copy link

@damascenorakuten damascenorakuten commented Feb 21, 2019

Same issue here:

panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
@dhanvi
Copy link

@dhanvi dhanvi commented Mar 21, 2019

kubectl port-forward sealed-secrets-controller-xxxxx 8080:8080 and then
curl -O localhost:8080/v1/cert.pem is doing the trick for me! from then on use the local file to seal secrets

@alejandrox1
Copy link

@alejandrox1 alejandrox1 commented Apr 2, 2019

In case anyone else comes across this issue, besides all the tremendously useful answers above, here is a practical summary of what to look for in order to get this going:
So #126 seems to have worked, however, this is NOT included in the v0.7.0 release of kubeseal (see commit history). So to have kubeseal --fetch-cert work, you may need to clone the repo and make kubeseal.

The second catch, is that kubeseal --fetch-cert uses a Kubernetes service to communicate with the sealed-secrets controller and get the cert from /v1/cert.pem (see comment right above this one). If for some reason you get an Error fetching certificate: services "sealed-secret-controller" not found then your service may be named something else (i.e., sealed-secrets).

So in short, you may have to clone the repo, make kubeseal, use that executable instead of the one in the release and make sure you are using the right service name.
In my case, I used the official Helm chart to deploy sealed-secrets and ended up with this:

$ kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   172.20.0.10     <none>        53/UDP,53/TCP   15h
sealed-secrets   ClusterIP   172.20.241.81   <none>        8080/TCP        14h
tiller-deploy    ClusterIP   172.20.235.94   <none>        44134/TCP       14h

So I had to do

$ ./kubeseal --fetch-cert --controller-name sealed-secrets
-----BEGIN CERTIFICATE-----
...

to get the cert.

@alejandrox1
Copy link

@alejandrox1 alejandrox1 commented Apr 2, 2019

As an extra comment, I got so involved in debugging this that I forgot what the original issue is:
Originally, if I wanted to do seal a secret, I got the following

$ kubeseal  <mysecret.json >mysealedsecret.json
panic: Error fetching certificate: services "http:sealed-secrets:" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9

Again, by cloning and compiling the kubeseal binary myself (again, this inlcudes the bump to v0.7.0 of client-go) I was the able to seal a secret as normal:

$ ./kubeseal --controller-name sealed-secrets <mysecret.json >mysealedsecret.json
@mbelang
Copy link

@mbelang mbelang commented Apr 11, 2019

@alejandrox1 When do we expect an official release?

@jgrabenstein
Copy link

@jgrabenstein jgrabenstein commented May 1, 2019

Would be pretty cool if there could be a new release cut with #126 included to resolve the issues of fetching the cert w/kubectl proxy. This also affects encrypting secrets (must download cert locally- and --fetch-cert is broken too). Last release was over 1 year ago. Cloning the repo and running make kubeseal does not work due to missing go dependencies.

For reference, this is the error I received:
panic: Error fetching certificate: an error on the server ("Error: 'dial tcp 10.65.1.2:8080: i/o timeout'\nTrying to reach: 'http://10.65.1.2:8080/v1/cert.pem'") has prevented the request from succeeding (get services http:sealed-secrets-controller:)

I'm now writing a workaround into my wrapper to use port-forwarding to fetch the cert, save it locally, use it for encryption, then remove the cert, instead of just being able to run kubeseal <mysecret.json >mysealedsecret.json

This did work as expected when I was using a cluster with Public IP's. Now that I am using a cluster with Private IP's, it is not, and I have to do a couple extra steps.

@alejandrox1
Copy link

@alejandrox1 alejandrox1 commented May 3, 2019

@mbelang I don't know 😅 I'm not actually a maintainer of this project.

@alejandrox1
Copy link

@alejandrox1 alejandrox1 commented May 3, 2019

@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.

@burnjake
Copy link

@burnjake burnjake commented Jun 27, 2019

Getting this when running on EKS Kubernetes version 1.12.

kubeseal --fetch-cert --token $(aws-iam-authenticator token -i my-cluster | jq -r '.status.token')
panic: Error fetching certificate: Unauthorized

goroutine 1 [running]:
main.main()
	/private/tmp/kubeseal-20181221-52005-1n89slr/sealed-secrets-0.7.0/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3ad

Does anyone have an update?

@JessieAMorris
Copy link

@JessieAMorris JessieAMorris commented Jun 27, 2019

If you use the git repo from master it works. They haven't done a release in a long time. Is this project still being maintained?

@jgrabenstein
Copy link

@jgrabenstein jgrabenstein commented Jun 27, 2019

@jgrabenstein do you use kubeseal the same way you use kubectl (or any other Kubernetes client)?
Could you provide more details about how your cluster is setup? Source code if possible to try and replicate it.

These are regional, private (no external/public IP's) GKE clusters on a shared VPC.
It appears that running simple commands like kubeseal <mysecret.json >mysealedsecret.json do not work with private clusters (see the error I reported above).

To work around that, I use kubectl port-forward to make the controller's port available locally. Then i can download the cert from the controller. Next you will encrypt with kubeseal, pointing it to the cert you just downloaded: kubeseal --cert /path/to/public-cert $RAW_SECRET > $SEALED_SECRET. You can then apply the sealed secret as you normally would.

Not sure my company would appreciate me sharing the source code for my wrapper.. If people here think it would help them I could possibly see if we could contribute it.

I think ultimately this project needs some decision/leadership around its future - this is something that could be fixed in kubeseal, even if it cannibalizes my wrapper's method.

@mkmik mkmik added this to the v0.8.0 milestone Jul 19, 2019
@mkmik
Copy link
Collaborator

@mkmik mkmik commented Jul 19, 2019

going to cut a new release soon

@mkmik mkmik closed this Jul 19, 2019
@jwenz723
Copy link

@jwenz723 jwenz723 commented Aug 3, 2019

Due to the fact that I have Weave CNI installed in my EKS cluster I seem to have broken the ability for kubectl proxy to be able to communicate to pods in my cluster. So if I run kubectl proxy and then browse to http://localhost:8001/api/v1/namespaces/kube-system/services/sealed-secrets-controller/proxy/healthz I get the following response:

Error: 'Address is not allowed'
Trying to reach: 'http://10.32.0.3:8080/healthz'

It seems like kubeseal --fetch-cert is relying on this functionality to work. The error from kubeseal I get is:

$ kubeseal --fetch-cert
panic: Error fetching certificate: the server is currently unable to handle the request (get services http:sealed-secrets-controller:)

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:340 +0x5c6

Currently the work around that I have gotten to work is:

$ kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080
$ curl -LJO http://localhost:8080/v1/cert.pem

This can also be done in a single shell using something like the following. This was useful to me as I am trying to retrieve the public key within a CI pipeline.

# inspired by https://stackoverflow.com/a/54931055
kubectl port-forward -n kube-system svc/sealed-secrets-controller 8080 &
proxy_pid=$!
echo $proxy_pid

until curl -fsSL http://localhost:8080/healthz > /dev/null; do
    echo "waiting for kubectl port-forward" >&2
    sleep 2
done

curl -LJO http://localhost:8080/v1/cert.pem

function cleanup {
    echo "killing kubectl proxy" >&2
    kill $proxy_pid
}
trap cleanup EXIT

Just posting this here in case it is useful to anyone else.

@mkmik
Copy link
Collaborator

@mkmik mkmik commented Aug 3, 2019

Would it make sense if the sealedsecret controller posted the public key into a configmap?

alinefr added a commit to alinefr/terraform-kubernetes-get-secret that referenced this issue Sep 22, 2020
EKS authentication needs an updated client-go version
Ref: bitnami-labs/sealed-secrets#110
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet