Kubelet TLS Bootstrap

[philips]

Feature Description

• One-line feature description (can be used as a release note): kubelet generates a private key and a CSR for submission to a cluster-level certificate signing process.
• Primary contact (assignee): @mikedanese
• Responsible SIGs: sig-auth
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @liggitt
• Approver (likely from SIG/area to which feature belongs): @liggitt
• Feature target (which target equals to which milestone):
  • Alpha release target (1.4)
  • Beta release target (1.6)
  • Stable release target (1.12)

[philips]

cc @kubernetes/sig-node FYI about this feature for Kubelet TLS bootstrap

[mikedanese]

I think we should consider this pre-alpha until we complete:

• kubectl integration
• kubelet integration
• e2e testing

Is anyone currently working on this?

[philips]

@mikedanese @gtank should be working on all of those things to get the feature done done.

[gtank]

Current status of what needs to be done:

1. kubelet needs to use the Certificates API. This will involve one new flag, provisionally --bootstrap-auth-token which will take a long random string. The kubelet uses this token to authenticate its requests to the Certificates API. Cluster-level access control should ensure that it can only be used for certificates requests. When this token exists, if the kubelet can't find TLS assets locally it will generate a fresh keypair and a Certificate Signing Request (CSR) to submit to the API. It will then watch the CSR object for the appearance of an issued certificate before proceeding with registration. It will use the certificate for client cert auth in subsequent API requests.
2. kubectl needs to fully support CSR objects. This is currently blocked on a problem with the swagger generation and base64-encoded []byte fields. We also need to document the best way of handling the manual approval flow.
3. Still need to write tests and update all the docs.

[philips]

Status update:

PR for kubelet TLS bootstrap is up: kubernetes/kubernetes#30094

[mikedanese]

Kubectl support tracked in kubernetes/kubernetes#30163 with some basic support prs out

[gtank]

Status update for TLS work:

In flight:

• Implement TLS bootstrap for kubelet using --bootstrap-auth-token Implement TLS bootstrap for kubelet using --bootstrap-auth-token kubernetes#30094
• Add an option to controller-manager to auto approve all CSRs add an option to controller-manager to auto approve all CSRs kubernetes#30153
• Add a certificate signing request resource printer in kubectl add a certificate signing request resource printer in kubectl kubernetes#30166
• Implement kubectl procelain csr commands implement kubectl procelain csr commands kubernetes#30237
• csr: add approval to the typed client csr: add approval to the typed client kubernetes#30236
• add shortname for certificate signing request in kubectl add shortname for certificate signing request in kubectl kubernetes#30165
• swagger: report golang type []byte as json type string swagger: report golang type []byte as json type string emicklei/go-restful#311

Issues:

• certificatesigningrequests should overwrite spec.user info in createstrategy certificatesigningrequests should overwrite spec.user info in createstrategy kubernetes#30239
• kubectl should support CertificateSigningRequest kubectl should support CertificateSigningRequest kubernetes#30163

Merged:

• add delete strategy to certificates registry add delete strategy to certificates registry kubernetes#30181

[liggitt]

It will use the certificate for client cert auth in subsequent API requests.

The scope of kubernetes/kubernetes#20439 was around obtaining TLS serving certs. The kubelet's use of this API should probably be limited to obtaining serving certs initially. Since the kubelet already needs API credentials to submit the initial CSR, obtaining client credentials this way doesn't buy us a whole lot from a bootstrapping perspective.

[mikedanese]

It allows us to bootstrap individual kubelet identities from a single cluster wide shared secret (bearer token for a kubelet-bootstrap user authorized to submit CSRs).

[liggitt]

If the goal is to identify kubelets individually so we can partition node permissions (which I am in favor of), multiple identities obtained from a single shared credential aren't meaningful from an auth perspective.

If we just want to identify requests from different nodes for debugging purposes, we can do that with user-agent strings (the same way controllers set up their own client user-agent string)

[mikedanese]

multiple identities obtained from a single shared credential

I'm not convinced of this. In fact I think this would work equally well if the CSR API was open (didn't require auth[n/z]). The shared secret is a mechanism that prevents a specific dos attack where a user could spam the CSR API with CSRs that would not be approved. @gtank can you weigh in on this?

[bgrant0607]

cc @kubernetes/sig-cluster-lifecycle

[philips]

@mikedanese @liggitt can we please move the technical discussion about the design of the system to the https://groups.google.com/forum/#!forum/kubernetes-sig-node mailing list.

[gtank]

@liggitt @mikedanese The nodes derive individual identities from the shared token. The token only exists to 1) filter requests meant for other clusters and 2) allow us restrict the nodes (via groups or ABAC) from accessing the rest of the API before they have client certs.

[pwittrock]

@philips Is this on target for the 1.4 feature freeze this Friday (Aug 19)?

[mike-saparov]

@pwittrock: based on kubernetes/kubernetes#30094 this feature suddenly started being redesigned in flight after 5 months of work... Any suggestions on how to avoid the major redesign three days before feature freeze are welcome

[smarterclayton]

Occasionally we will realize gaps in designed features as they are implemented. We definitely need a process that handles that - rushing features to delivery without all the proper technical due diligence being done is going to cause more issues than occasionally features missing a release. I think everyone involved is trying to find the best and most secure option for the platform here.

[mike-saparov]

@smarterclayton based on PR discussions it would be great if @liggitt and @deads2k could help with parts of the feature, wdyt? any coding / review help is appreciated to address raised concerns.

[smarterclayton]

I believe both of them had been representing sig-auth on the security aspects here - I had not yet seen an update on the remaining issues though. That's probably appropriate on the linked issue.

[justaugustus]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-node-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y)
  • Stable release target (x.y)

Set the following:

• Description
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[luxas]

@kubernetes/sig-auth-feature-requests can we graduate this to stable in v1.12?

[justaugustus]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-auth-misc --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

[mikedanese]

I'm ok with this going to GA as long as the certificates API remains beta.

[liggitt]

Can a GA feature depend on a beta API? What does that mean, support-wise?

[justaugustus]

I've added this to the 1.12 sheet as stable, but let me know if we need to walk that back.
cc: @kacole2 @wadadli @robertsandoval @rajendar38

[luxas]

I'm ok with this going to GA as long as the certificates API remains beta.

Why not graduate the as-is API to GA and do a v2 in case we need additional features or whatever?

[liggitt]

We discussed this in the sig-auth meeting on 8/8, and agreed the bootstrap feature can be promoted to stable, but not the CSR API yet. That means the externally-facing portions of the bootstrap mechanism (bootstrap kubeconfig, etc) will continue to be supported.

[zparnold]

Hey there! @philips I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[philips]

Zach- I filed the feature initially but other folks took over the work. So, I won't be able to help with the docs. Sorry.

…

On Mon, Aug 20, 2018 at 2:25 PM Zach Arnold ***@***.***> wrote: Hey there! @philips <https://github.com/philips> I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACDCOhjX6sz-ErOXML9lrqSGnd4eHTUks5uSykwgaJpZM4JS7ez> .

[zparnold]

Ok, thanks for letting me know. Whom shall I ask for this data then in your opinion? On Mon, Aug 20, 2018 at 2:26 PM Brandon Philips <notifications@github.com> wrote:

…

Zach- I filed the feature initially but other folks took over the work. So, I won't be able to help with the docs. Sorry. On Mon, Aug 20, 2018 at 2:25 PM Zach Arnold ***@***.***> wrote: > Hey there! @philips <https://github.com/philips> I'm the wrangler for the > Docs this release. Is there any chance I could have you open up a docs PR > against the release-1.12 branch as a placeholder? That gives us more > confidence in the feature shipping in this release and gives me something > to work with when we start doing reviews/edits. Thanks! If this feature > does not require docs, could you please update the features tracking > spreadsheet to reflect it? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#43 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AACDCOhjX6sz-ErOXML9lrqSGnd4eHTUks5uSykwgaJpZM4JS7ez > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE81SF57z1mjfKYZ9QPBQVJHzoItTVRYks5uSymggaJpZM4JS7ez> .

[mikedanese]

@zparnold please use the assignee of the feature issue rather than the original creator (which is immutable) of the issue for this kind of stuff. I can do this.

[zparnold]

Ok, thanks! Learning the ropes around here. :)

…

On Mon, Aug 20, 2018 at 2:37 PM Mike Danese ***@***.***> wrote: @zparnold <https://github.com/zparnold> please use the assignee of the feature issue rather than the original creator (which is immutable) of the issue for this kind of stuff. I can do this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE81SHodJlKFUOZaBgWmnDEiCYq5HUr6ks5uSywMgaJpZM4JS7ez> .

[zparnold]

@mikedanese Will you be able to handle docs?

[justaugustus]

@mikedanese --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

[mikedanese]

GA docs PR here:

kubernetes/website#10232

[kacole2]

It looks like this has graduated to GA so I'm going to clean up and close the issue. nice job everyone!

/close

[k8s-ci-robot]

@kacole2: Closing this issue.

In response to this:

It looks like this has graduated to GA so I'm going to clean up and close the issue. nice job everyone!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.