New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When bootstrapping a client cert, store it with other client certs #62152

Merged
merged 1 commit into from Apr 23, 2018

Conversation

Projects
None yet
4 participants
@smarterclayton
Contributor

smarterclayton commented Apr 5, 2018

The kubelet uses two different locations to store certificates on
initial bootstrap and then on subsequent rotation:

  • bootstrap: certDir/kubelet-client.(crt|key)
  • rotation: certDir/kubelet-client-(DATE|current).pem

Bootstrap also creates an initial node.kubeconfig that points to the
certs. Unfortunately, with short rotation the node.kubeconfig then
becomes out of date because it points to the initial cert/key, not the
rotated cert key.

Alter the bootstrap code to store client certs exactly as if they would
be rotated (using the same cert Store code), and reference the PEM file
containing cert/key from node.kubeconfig, which is supported by kubectl
and other Go tooling. This ensures that the node.kubeconfig continues to
be valid past the first expiration.

Example:

bootstrap:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem
  writes node.kubeconfig pointing to certDir/kubelet-client-current.pem
rotation:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem

This will also allow us to remove the wierd "init store with bootstrap cert" stuff, although I'd prefer to do that in a follow up.

@mikedanese @liggitt as per discussion on Slack today

The `--bootstrap-kubeconfig` argument to Kubelet previously created the first bootstrap client credentials in the certificates directory as `kubelet-client.key` and `kubelet-client.crt`.  Subsequent certificates created by cert rotation were created in a combined PEM file that was atomically rotated as `kubelet-client-DATE.pem` in that directory, which meant clients relying on the `node.kubeconfig` generated by bootstrapping would never use a rotated cert.  The initial bootstrap certificate is now generated into the cert directory as a PEM file and symlinked to `kubelet-client-current.pem` so that the generated kubeconfig remains valid after rotation.
@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 5, 2018

/test pull-kubernetes-verify

1 similar comment
@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 5, 2018

/test pull-kubernetes-verify

@mikedanese

This comment has been minimized.

Member

mikedanese commented Apr 6, 2018

We've noticed weird corruptions of the kubeconfig after a bootstrap, e.g. "server.go:140] tls: failed to find any PEM data in certificate input". I wonder if the recovery in filestore will do better then the old LoadClientCert.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 6, 2018

The atomic swap should help.

}
}
}()
}
if keyData == nil {

This comment has been minimized.

@mikedanese

mikedanese Apr 9, 2018

Member

len(keyData) == 0. Should we do any other integrity checks here?

This comment has been minimized.

@smarterclayton

smarterclayton Apr 12, 2018

Contributor

Probably. Let me noodle it.

This comment has been minimized.

@smarterclayton

smarterclayton Apr 23, 2018

Contributor

Updated with sanity checking that it's a valid private key at minimum.

@mikedanese

This comment has been minimized.

Member

mikedanese commented Apr 9, 2018

One comment, otherwise LGTM.
/approve

When bootstrapping a client cert, store it with other client certs
The kubelet uses two different locations to store certificates on
initial bootstrap and then on subsequent rotation:

* bootstrap: certDir/kubelet-client.(crt|key)
* rotation:  certDir/kubelet-client-(DATE|current).pem

Bootstrap also creates an initial node.kubeconfig that points to the
certs. Unfortunately, with short rotation the node.kubeconfig then
becomes out of date because it points to the initial cert/key, not the
rotated cert key.

Alter the bootstrap code to store client certs exactly as if they would
be rotated (using the same cert Store code), and reference the PEM file
containing cert/key from node.kubeconfig, which is supported by kubectl
and other Go tooling. This ensures that the node.kubeconfig continues to
be valid past the first expiration.
@mikedanese

This comment has been minimized.

Member

mikedanese commented Apr 23, 2018

@mikedanese

This comment has been minimized.

Member

mikedanese commented Apr 23, 2018

/lgtm

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Apr 23, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikedanese, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 23, 2018

/retest

1 similar comment
@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 23, 2018

/retest

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Apr 23, 2018

Automatic merge from submit-queue (batch tested with PRs 63001, 62152, 61950). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit 939c078 into kubernetes:master Apr 23, 2018

15 checks passed

Submit Queue Queued to run github e2e tests a second time.
Details
cla/linuxfoundation smarterclayton authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment