Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide full secret rotation #1020

Open
justinsb opened this issue Nov 30, 2016 · 26 comments

Comments

@justinsb
Copy link
Member

commented Nov 30, 2016

We should provide a way to rotate all our secrets: usernames, tokens, CA key, SSH key.

@kris-nova

This comment has been minimized.

Copy link
Member

commented Dec 11, 2016

@justinsb

Curious do we want 1 command to rotate them all, or do we need individual execution paths for all of our secrets?

Should we also include a way to generate these secrets within kops automatically?

@chrislovecnm

This comment has been minimized.

Copy link
Member

commented Dec 11, 2016

@kris-nova I am thinking that we do a rolling update of the cluster. All components to need a restart and the kubeconfig is going to change. I am pretty sure kops already generates the secrets, we just need to be able to roll and update. Also we need to plugin to 3rd part cert sources.

Secretes:

  • TLS Certs
  • Admin ssh key
  • Admin password

^ What am I missing??

@kris-nova

This comment has been minimized.

Copy link
Member

commented Dec 11, 2016

I was hoping kops could generate an ssh key for the user if they wanted one ad-hoc - do we do this yet?

@chrislovecnm

This comment has been minimized.

Copy link
Member

commented Dec 17, 2016

Nope we require an ash key

@justinsb justinsb added this to the 1.5.1 milestone Dec 28, 2016

@lev-kuznetsov

This comment has been minimized.

Copy link

commented Jul 11, 2017

Is this still on a roadmap somewhere?

Is there a workaround I can follow in the meantime? We need a procedure for rotating credentials.

@chrislovecnm

This comment has been minimized.

Copy link
Member

commented Jul 12, 2017

At this point creating a new secret and applying acrolling update is your option.

@fejta-bot

This comment has been minimized.

Copy link

commented Dec 31, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

commented Jan 30, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 1, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@Christian-Schmid

This comment has been minimized.

Copy link

commented Mar 7, 2019

/remove-lifecycle rotten
/reopen

Hi guys,

I'm opening this issue again, as we have the same challenge currently.
As the issue is already a bit old, I want to ask, if by now maybe something in that direction happend.
I found a sketched procedure here: https://github.com/kubernetes/kops/blob/master/docs/rotate-secrets.md but this one does not work without downtime.

Does maybe someone got some ideas / experiences how to solves this?

Thanks a lot!

Chris

@Christian-Schmid

This comment has been minimized.

Copy link

commented Mar 7, 2019

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

@Christian-Schmid: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@philwhln

This comment has been minimized.

Copy link

commented Mar 25, 2019

Does maybe someone got some ideas / experiences how to solves this?

@Christian-Schmid Did you find any non-downtime solutions to this? We're looking at the same thing now.

@mikesplain

This comment has been minimized.

Copy link
Member

commented Mar 26, 2019

On behalf of @Christian-Schmid

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@mikesplain: Reopened this issue.

In response to this:

On behalf of @Christian-Schmid

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Mar 26, 2019

@Christian-Schmid

This comment has been minimized.

Copy link

commented Mar 26, 2019

Hi @philwhln
We're still looking for a "nice" solution to rotate the secrets.
One reason why we want to do the rotation is that we want to control external access to the kubernets api.
One option we were evaluating was to put an nginx reverse proxy in front of the kube api.
Like this we wouldn't really have to rotate the certificates, and still could control the external api access with other means of authentication.

But if this workaround helps, depends on the use case why you want to rotate :-)

@philwhln

This comment has been minimized.

Copy link

commented Mar 26, 2019

Hi @Christian-Schmid ,

In the short-term, we're looking to rotate out the keys we used in early development of our clusters as these were used by people who have left the company. In the longer-term, we see this as good practice. Interesting idea with the reverse proxy. Would this work?

We're not overly confident in https://github.com/kubernetes/kops/blob/master/docs/rotate-secrets.md since it seems to have been written 18 months ago with little review or updates. That said, we're going to dig into it and test it out. @justinsb, I'm interested in your thoughts on this, since you wrote that doc and also opened this ticket :)

@Christian-Schmid

This comment has been minimized.

Copy link

commented Mar 29, 2019

Regarding the rotate-secrets manual: we tested the described steps and it more or less worked as described with a downtime of like 15 minutes (due to we had to force cluster update twice).
The only step which we had a problem was the line:

kops get secrets | grep ^Keypair | awk '{print $2}' | xargs -I {} kops delete secret keypair {}

which caused an error in the current kops version. But we could delete the pki directly from the s3 bucket with aws cli:
aws s3 rm s3://<your_bucket>.com/pki/issued --recursive
aws s3 rm s3://<your_bucket>.com/pki/private --recursive

Regarding proxy: We didn't investigate so far further in that direction due to time reasons...

@philwhln

This comment has been minimized.

Copy link

commented Mar 29, 2019

@Christian-Schmid . Thanks for this info!

The only step which we had a problem was the line:

We hit the same problem and decided not proceed. Good to know that deleting in S3 worked and you were able to complete the process. We had considered this too. Downtime is not great though :)

@tushar00jain

This comment has been minimized.

Copy link

commented Apr 9, 2019

+1

@tushar00jain

This comment has been minimized.

Copy link

commented May 2, 2019

@justinsb

It doesn't look like kubernetes supports the usage of multiple certs that would make a zero downtime rotation possible. I would appreciate if you could point me to the sig responsible for pki or maybe point me on if and how I could start working on this myself? This issue seems pretty critical and judging by the response on it doesn't seem likely that waiting for 10 years for someone to implement a solution would be sufficient. If somehow our secrets get leaked then also there is no way to rotate the certificates unless we accept the hefty downtime.

@chrislovecnm

At this point creating a new secret and applying acrolling update is your option.

Could you explain what you mean by this? You mean the current documented method that involves deleting all pki related data on S3 or something else?

@tushar00jain

This comment has been minimized.

Copy link

commented May 2, 2019

kubernetes/kubeadm#581
kubernetes/kubeadm#1361

Maybe kubernetes and kubeadm supports renewal but lacks the docs, certainly for HA master nodes judging by the comments on the above mentioned issues. After those are addressed maybe the parts of the code can be also ported over from kubeadm.

@tushar00jain

This comment has been minimized.

Copy link

commented May 4, 2019

BTW @philwhln @Christian-Schmid for your use cases about external access, maybe the best approach is to use OIDC or something like AWS authenticator instead of x509. There's good support for these approaches already.

And one way to rotate the certificates with minimum downtime would be to spin up a warm standby cluster when the certificates are about to expire and move over the traffic to this other cluster.

@philwhln

This comment has been minimized.

Copy link

commented May 4, 2019

use OIDC

@tushar00jain We already do use this, with dex (similar to https://thenewstack.io/kubernetes-single-sign-one-less-identity/), but this doesn't remove the need for Kubernetes to have certificates that need rotating.

move over the traffic to this other cluster.

This does seem like the only solution right now, but there's a cost to this and something we don't think we should have to do.

@tushar00jain

This comment has been minimized.

Copy link

commented May 4, 2019

but this doesn't remove the need for Kubernetes to have certificates that need rotating.

yes at least the downtime should be in seconds if at all but the approach mentioned by deleting the pki folder on S3 is just not feasible

@fejta-bot

This comment has been minimized.

Copy link

commented Aug 2, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.