Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm: enhanced TLS validation for token-based discovery in `kubeadm join` #49520

Merged
merged 3 commits into from Aug 10, 2017

Conversation

@mattmoyer
Copy link
Member

mattmoyer commented Jul 24, 2017

What this PR does / why we need it:
This PR implements enhanced TLS validation for kubeadm join when using token-based TLS discovery. Without this enhancement, kubeadm join has some less-than-ideal security properties. Specifically, in the case where a bootstrap token is compromised, the attacker can impersonate the API server to newly bootstrapping clients (more discussion in the design proposal).

The gist of this enhancement is to support public key pinning in the style of RFC7469. When bootstrapping, kubeadm can now be configured with a whitelist of root CA public keys. It can then validate that the cluster it connects to is operated by the owner of one of those public keys.

These public key hashes are short enough that the entire kubeadm join command can still be copy-pasted relatively easily (not as easily as before, but ~160 characters). Using a public key hash rather than a hash over the entire certificate allows certificates to be reissued with updated expirations without invalidating existing key pins.

This change adds two new command line flags (and associated config parameters):

  • --discovery-token-ca-cert-hash sha256:<hash>:
    Validates that the cluster root CA has a public key fingerprint that matches one of the specified values. If this flag is not passed when token-based discovery is being used, a warning is printed. This warning will become an error in 1.9.
  • --discovery-token-unsafe-skip-ca-verification:
    Disables the warning message when no keys are pinned. In 1.9, this flag will be required unless --discovery-token-unsafe-skip-ca-verification is used.

This is fully backwards compatible and client side (kubeadm) only. It will be a breaking change when the flag becomes required in v1.9.

This validation is done after and in addition to the existing bootstrap token signing/MAC mechanism.

Example from kubeadm init:

$ kubeadm init
[...]

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token a66ae0.1f8a5ed9a210e187 192.168.42.10:6443 --discovery-token-ca-cert-hash sha256:547c102383c0f26387b961b4e9b8f842dc07c074c8316f238dbcf5563fc3ac35

Example from kubeadm join:

$ kubeadm join --token a66ae0.1f8a5ed9a210e187 192.168.42.10:6443 --discovery-token-ca-cert-hash sha256:547c102383c0f26387b961b4e9b8f842dc07c074c8316f238dbcf5563fc3ac35
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.42.10:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.42.10:6443"
[discovery] Requesting info from "https://192.168.42.10:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.42.10:6443"
[discovery] Successfully established connection with API Server "192.168.42.10:6443"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

Which issue this PR fixes:
ref kubernetes/enhancements#130
fixes: kubernetes/kubeadm#365

Special notes for your reviewer:
This was proposed and discussed briefly by SIG-cluster-lifecycle and SIG-auth. The design proposal is in Google Docs.

There is a documentation change needed to explain the security properties of kubeadm join with and without --discovery-token-ca-cert-hash. This page should be linked by to by the warning message when you don't pass either of the new flags (I have it pointing here for now, which I think will be the right place). I will follow up with this documentation shortly.

Release note:

kubeadm: added enhanced TLS validation for token-based discovery in `kubeadm join` using a new `--discovery-token-ca-cert-hash` flag.

/cc @luxas @jbeda @ericchiang

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jul 24, 2017

Hi @mattmoyer. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ericchiang

This comment has been minimized.

Copy link
Member

ericchiang commented Jul 24, 2017

/ok-to-test

High level opinion that I brought up in the design doc, but wanted to reiterate here.

The value of --tls-discovery-root-ca for kubeadm join can't be known until kubeadm init runs on the master to generate the CA. This means I can't provision my workers until my masters have been provisioned and kubeadm init has run on the master

This won't work for anyone who provisions the masters and the workers at the same time. For example, Kubernetes Anywhere would have to use the --tls-discovery-allow-unsafe:

https://github.com/kubernetes/kubernetes-anywhere/blob/8d2c7788c63693b0021278dedcf03fad935efef1/phase1/gce/configure-vm-kubeadm.sh#L40

I am for this fix, but I think a majority of auto-scaling setups will have to use the unsafe flag.


// Set is a set of pinned x509 public keys.
type Set struct {
sha256Hashes [][]byte

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

This can probably just be map[string]bool. You're always passing around the string value anyway.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

+1, will fix. I originally had this code supporting other hash types/lengths (truncated SHA-256/128 and SHA-256 encoded as base58). In the end I just left plain hex-encoded SHA-256 so this doesn't make a lot of sense.

type certChain []*x509.Certificate

// saveCertificateChain parses and saves the array of DER-encoded certificates into a certChain
func (c *certChain) saveCertificateChain(rawCerts [][]byte, _ [][]*x509.Certificate) error {

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

Why do you need to save the certificate chain? Don't you just need the CA that's part of the cluster-info?

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

I think Matt's intention here is to make sure the server presents the same CA key the first time we're contacting it insecurely as after, when we have validated the authenticity of the cluster-info ConfigMap and actually have a CA cert to trust.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Yeah, validating the entire initial connection gives better trust in the other parts of the cluster-info ConfigMap since we use it for more than just the cluster CA. I think we could also accomplish this by making two requests if we want to go that route.

We need the full chain since the server might have some intermediate CAs included that we need to chain through to get from the root to the leaf cert.

var certificates certChain

// Create an HTTP client with a custom TLS transport that saves off the certificate chain for later validation
insecureClient := http.Client{

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

We had issues constantly re-initializing transports like this since they have a tendency to keep TCP connections alive.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

That makes sense. I think it shouldn't be a huge issue for kubeadm since it's relatively short-lived. That's still pretty non-ideal, though.

I'll see if I can find a way to rework this so the http.Client is long lived.


// Make sure the wrapper is being used correctly
if len(certificates) == 0 {
return nil, nil, errInsecureClientNoRequest

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

If you error here you won't close the response body.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Ah, thanks. Will fix.

tokenutil "k8s.io/kubernetes/cmd/kubeadm/app/util/token"
bootstrapapi "k8s.io/kubernetes/pkg/bootstrap/api"
"k8s.io/kubernetes/pkg/controller/bootstrap"
)

const BootstrapUser = "token-bootstrap-client"

var (

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

Does kubeadm normally declare errors like this? I'd expect the fmt.Errorf call to just be inlined. e.g.

return fmt.Errorf("this version of kubeadm only supports deploying clusters with the control plane version >= %s. Current version: %s", kubeadmconstants.MinimumControlPlaneVersion.String(), cfg.KubernetesVersion)

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

No we haven't done this anywhere else.
I see it's used for testing.
I don't have a strong opinion here, though.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Yeah, this was mainly so I can test that particular error cases return the correct error.

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

I'm ok with it, but we aren't doing it in other places. This seems to be a good place to use this method though...

defer response.Body.Close()

// Read the entire response body
responseBytes, err := ioutil.ReadAll(response.Body)

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

You don't trust the server at this point. Probably good practice to limit the number of bytes read to something reasonable.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

+1 will fix.

chainCerts.AddCert(chainCert)
}

// Now that we know the root CA pool, validate the original certificate chain

This comment has been minimized.

@ericchiang

ericchiang Jul 24, 2017

Member

It seems much simpler and less prone to error to just make a second request to get the cluster-info once you have the CA and let the Go TLS library automatically do the validation.

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

+1, agree

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Making double requests seemed worse to me, but I think it's sound from a security perspective.

}

// TODO: convert this warning to an error after v1.8
if len(cfg.DiscoveryFile) == 0 && len(cfg.TLSDiscoveryRootCAPubKeys) == 0 && !cfg.TLSDiscoveryAllowUnsafe {

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

Do we care about if DiscoveryFile is unset and/or DiscoveryToken is set here or can this be simplified to just check PubKeys and AllowUnsafe?

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

The way it's written now, --tls-discovery-root-ca only applies to the plain token-based case. If you're using DiscoveryFile with local files or a remote HTTPS endpoint, it doesn't do any validation.

I think that's reasonable because DiscoveryFile should be transmitted through some secure means that protects its integrity. Either the local file will have been copied there by some secure orchestrator or the remote file will be authenticated by the web PKI and the system roots. I don't think we need to consider these cases unsafe or require the user to pass --tls-discovery-allow-unsafe.

I have the validation error TLSDiscoveryRootCAPubKeys cannot be used with DiscoveryFile to make sure you don't get a false sense of security from thinking you've pinned the CA when you're actually just relying on the security of the discovery file.

tokenutil "k8s.io/kubernetes/cmd/kubeadm/app/util/token"
bootstrapapi "k8s.io/kubernetes/pkg/bootstrap/api"
"k8s.io/kubernetes/pkg/controller/bootstrap"
)

const BootstrapUser = "token-bootstrap-client"

var (

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

No we haven't done this anywhere else.
I see it's used for testing.
I don't have a strong opinion here, though.


// For each cluster, validate that its CA matches a pinned key and save that root into an x509.CertPool
pinnedRoots := x509.NewCertPool()
for _, cluster := range finalConfig.Clusters {

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

The spec states that there must only be one cluster in the cluster-info kubeconfig file.
Basically that single Cluster's CA cert and server endpoint is everything we're using

This comment has been minimized.

@mattmoyer

mattmoyer Jul 26, 2017

Author Member

Thanks. Instead of looping I added an assertion that there is only a single cluster.

if err != nil {
return nil, err
return nil, fmt.Errorf("invalid URL, can't connect: %v", err)

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

suggest: invalid URL %q, ...

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Thanks, will fix.

chainCerts.AddCert(chainCert)
}

// Now that we know the root CA pool, validate the original certificate chain

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

+1, agree

type certChain []*x509.Certificate

// saveCertificateChain parses and saves the array of DER-encoded certificates into a certChain
func (c *certChain) saveCertificateChain(rawCerts [][]byte, _ [][]*x509.Certificate) error {

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

I think Matt's intention here is to make sure the server presents the same CA key the first time we're contacting it insecurely as after, when we have validated the authenticity of the cluster-info ConfigMap and actually have a CA cert to trust.

// which should perform a single HTTP request and return the response. The temporary
// client will not validate TLS certificates in real time, but instead will capture
// and return the certificates for post-hoc validation.
func withInsecureHTTPClient(

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

Is there no way to save the cert chain with the normal kube client?
So far we've been able to just skip TLS verification with the normal kube client.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

We could extend the normal API client to give more control over the underlying http.Client. I think extending it to support this particular case wouldn't be worth it, but a more general change that allowed passing in a custom http.Client could be more generally useful.

The thinking with doing this as a "raw" http.Client instead of using the normal API client was that this is fairly simple from an HTTP standpoint (single GET request) and that the TLS bits were such an edge case that we probably don't want that complexity in the main API client code.


// Package pubkeypin provides primitives for x509 public key pinning in the
// style of RFC7469.
package pubkeypin

This comment has been minimized.

@luxas

luxas Jul 25, 2017

Member

Do we consider this kubeadm-specific or can we move this to pkg/util or something like k8s.io/apimachinery/pkg/util/sets?

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Right now kubeadm is the only thing that knows or cares about these public key pins. It might be useful somewhere else (maybe in kubectl also?).

I don't feel strongly about this either way.

This comment has been minimized.

@ericchiang

ericchiang Jul 25, 2017

Member

Please keep it in kubeadm. The more dependencies we create, the harder it's going to be to split the repos in the future.

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Jul 25, 2017

[discovery] Failed to connect to API Server "192.168.42.10:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "ec2651" is invalid for this cluster, can't connect

@mattmoyer Are you sure why this happened? Was the token actually not there or is it an error in this code?

The value of --tls-discovery-root-ca for kubeadm join can't be known until kubeadm init runs on the master to generate the CA. This means I can't provision my workers until my masters have been provisioned and kubeadm init has run on the master

@ericchiang The user still has the power to pre-provision the CA, so they can know what the CA hash will be in beforehand
Basically doing this:

// generate CA private key
openssl --foo -out /etc/kubernetes/pki/ca.key
// generate CA certificate
openssl --bar -out /etc/kubernetes/pki/ca.crt
// get CA hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>&/dev/null | openssl dgst -sha256 -hex > my-CA-hash.txt

kubeadm init --token <id>.<secret>
@ericchiang

This comment has been minimized.

Copy link
Member

ericchiang commented Jul 25, 2017

The user still has the power to pre-provision the CA, so they can know what the CA hash will be in beforehand

@luxas ah, great! Is this actual endorsed behavior or a nice consequence of the code? Might be worth updating the docs as part of this PR.

@ericchiang

This comment has been minimized.

Copy link
Member

ericchiang commented Jul 25, 2017

@mattmoyer

This comment has been minimized.

Copy link
Member Author

mattmoyer commented Jul 25, 2017

@luxas

[discovery] Failed to connect to API Server "192.168.42.10:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "ec2651" is invalid for this cluster, can't connect

@mattmoyer Are you sure why this happened? Was the token actually not there or is it an error in this code?

I think this was just an artifact of my local vagrant environment. It's not reproducible locally, so I think the server just wasn't up quite yet in this case.

@jbeda

jbeda approved these changes Jul 25, 2017

Copy link
Contributor

jbeda left a comment

LGTM.

I'm torn on the "validate current conn" vs. "make a new conn". I could go either way. So I won't express an opinion.

@@ -101,4 +101,16 @@ type NodeConfiguration struct {
NodeName string
TLSBootstrapToken string
Token string

// TLSDiscoveryRootCAPubKeys specifies a set of public key pins. The root CA
// discovered during TLS bootstrapping must match one of these values.

This comment has been minimized.

@jbeda

jbeda Jul 25, 2017

Contributor

Technically, this is during the bootstrap discovery, not TLS. We use this so that we can establish the kubelet trusting the master.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

Ah, yeah I'll fix that comment and see if I made the same mistake elsewhere. I originally approached this code without a clear understanding of the "discovery" vs. "TLS bootstrapping" terminology.


// TLSDiscoveryRootCAPubKeys specifies a set of public key pins. The root CA
// discovered during TLS bootstrapping must match one of these values.
// Specifying an empty set disables root CA pinning. Each hash is a hex encoded

This comment has been minimized.

@jbeda

jbeda Jul 25, 2017

Contributor

I'd still love to see us do something like base 36 or base 58 here to shorten this. But that can come in a later PR.

This comment has been minimized.

@mattmoyer

mattmoyer Jul 25, 2017

Author Member

I tried to implement the pubkeypin so it's easy to swap in support for multiple encodings and lengths.

I think if we do that we should also switch the default token encodings as well.

@mattmoyer

This comment has been minimized.

Copy link
Member Author

mattmoyer commented Jul 25, 2017

pull-kubernetes-kubemark-e2e-gce failure is #49320

@luxas luxas added this to the v1.8 milestone Jul 26, 2017

@timothysc
Copy link
Member

timothysc left a comment

going to recuse myself from this one other then to say, please squash commits.

@mattmoyer

This comment has been minimized.

Copy link
Member Author

mattmoyer commented Jul 26, 2017

I think the latest change addresses all the reviews so far, including ripping out the post-hoc validation in favor of just making two requests.

I'm leaving the original commits in https://github.com/mattmoyer/kubernetes/commits/8d9feb79c1619d8a80350f8ce1aa6dbd47350e89 and will squash/rebase this PR against current master for a final review.

Thanks for all the feedback.

@k8s-github-robot k8s-github-robot removed the lgtm label Aug 6, 2017

@mattmoyer

This comment has been minimized.

Copy link
Member Author

mattmoyer commented Aug 6, 2017

@mikedanese : Thanks for the review. I agree that the names weren't perfect. I just updated with the names you suggested, added an explicit type on the hash value (so it's sha256:<value>), and expanded the usage/docs a little.

I'll update the PR body with the new flags as well.

mattmoyer added some commits Jul 24, 2017

kubeadm: add pubkeypin package (public key pinning hash implementation).
This change adds a `k8s.io/kubernetes/cmd/kubeadm/app/util/pubkeypin` package which implements x509 public key pinning in the style of RFC7469. This is the public key hash format used by the new `kubeadm join --discovery-token-ca-cert-hash` flag.

Hashes are namespaced with a short type, with "sha256" being the only currently-supported format. Type "sha256" is a hex-encoded SHA-256 hash over the Subject Public Key Info (SPKI) object in DER-encoded ASN.1.
kubeadm: implement TLS discovery root CA pinning.
This change adds the `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join` and corresponding fields on the kubeadm NodeConfiguration struct. These flags configure enhanced TLS validation for token-based discovery.

The enhanced TLS validation works by pinning the public key hashes of the cluster CA. This is done by connecting to the `cluster-info` endpoint initially using an unvalidated/unsafe TLS connection. After the cluster info has been loaded, parsed, and validated with the existing symmetric signature/MAC scheme, the root CA is validated against the pinned public key set. A second request is made using validated/safe TLS using the newly-known CA and the result is validated to make sure the same `cluster-info` was returned from both requests.

This validation prevents a class of attacks where a leaked bootstrap token (such as from a compromised worker node) allows an attacker to impersonate the API server.

This change also update `kubeadm init` to print the correct `--discovery-token-ca-cert-hash` flag in the example `kubeadm join` command it prints at the end of initialization.
kubeadm: generated deepcopy for `k8s.io/kubernetes/cmd/kubeadm/app/ap…
…is/kubeadm` and `k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1`.

@mattmoyer mattmoyer force-pushed the mattmoyer:bootstrap-pinning branch from b17e3f8 to 358806e Aug 10, 2017

@mikedanese mikedanese added lgtm and removed do-not-merge labels Aug 10, 2017

@mikedanese

This comment has been minimized.

Copy link
Member

mikedanese commented Aug 10, 2017

/lgtm

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Aug 10, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ericchiang, luxas, mattmoyer, mikedanese

Associated issue: 130

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@mikedanese

This comment has been minimized.

Copy link
Member

mikedanese commented Aug 10, 2017

/retest

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Aug 10, 2017

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Aug 10, 2017

Automatic merge from submit-queue

@k8s-github-robot k8s-github-robot merged commit b42a7b3 into kubernetes:master Aug 10, 2017

9 of 10 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-verify
Details
cla/linuxfoundation mattmoyer authorized
Details
pull-kubernetes-bazel Job succeeded.
Details
pull-kubernetes-e2e-gce-etcd3 Jenkins job succeeded.
Details
pull-kubernetes-e2e-kops-aws Jenkins job succeeded.
Details
pull-kubernetes-federation-e2e-gce Jenkins job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Jenkins job succeeded.
Details
pull-kubernetes-node-e2e Jenkins job succeeded.
Details
pull-kubernetes-unit Jenkins job succeeded.
Details
pull-kubernetes-verify Jenkins job succeeded.
Details

@mattmoyer mattmoyer deleted the mattmoyer:bootstrap-pinning branch Aug 11, 2017

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 18, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520.

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 19, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520.

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 19, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 19, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 19, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

mattmoyer added a commit to mattmoyer/kubernetes.github.io that referenced this pull request Sep 22, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8.
This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

steveperry-53 added a commit to kubernetes/website that referenced this pull request Sep 22, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524
)

This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

steveperry-53 added a commit to kubernetes/website that referenced this pull request Sep 29, 2017

Release 1.8 (#5659)
* GC now supports non-core resources

* Add two examples about how to analysis audits of kube-apiserver (#4264)

* Deprecate system:nodes binding

* [1.8] StatefulSet `initialized` annotation is now ignored.

* inits the kubeadm upgrade docs

addresses /issues/4689

* adds kubeadm upgrade cmd to ToC

addresses /issues/4689

* add workload placement docs

* ScaleIO - document udpate for 1.8

* Add documentation on storageClass.mountOptions and PV.mountOptions (#5254)

* Add documentation on storageClass.mountOptions and PV.mountOptions

* convert notes into callouts

* Add docs for CustomResource validation

add info about supported fields

* advanced audit beta features (#5300)

* Update job workload doc with backoff failure policy (#5319)

Add to the Jobs documentation how to use the new backoffLimit field that
limit the number of Pod failure before considering the Job as failed.

* Documented additional AWS Service annotations (#4864)

* Add device plugin doc under concepts/cluster-administration. (#5261)

* Add device plugin doc under concepts/cluster-administration.

* Update device-plugins.md

* Update device-plugins.md

Add meta description. Fix typo. Change bare metal deployment to manual deployment.

* Update device-plugins.md

Fix typo again.

* Update page.version. (#5341)

* Add documentation on storageClass.reclaimPolicy (#5171)

* [Advanced audit] use new herf for audit-api (#5349)

This tag contains all the changes in v1beta1 version. Update it now.

* Added documentation around creating the InitializerConfiguration for the persistent volume label controller in the cloud-controller-manager (#5255)

* Documentation for kubectl plugins (#5294)

* Documentation for kubectl plugins

* Update kubectl-plugins.md

* Update kubectl-plugins.md

* Updated CPU manager docs to match implementation. (#5332)

* Noted limitation of alpha static cpumanager.

* Updated CPU manager docs to match implementation.

- Removed references to CPU pressure node condition and evictions.
- Added note about new --cpu-manager-reconcile-period flag.
- Added note about node allocatable requirements for static policy.
- Noted limitation of alpha static cpumanager.

* Move cpu-manager task link to rsc mgmt section.

* init containers annotation removed in  1.8 (#5390)

* Add documentation for TaintNodesByCondition (#5352)

* Add documentation for TaintNodesByCondition

* Update nodes.md

* Update taint-and-toleration.md

* Update daemonset.md

* Update nodes.md

* Update taint-and-toleration.md

* Update daemonset.md

* Fix deployments (#5421)

* Document extended resources and OIR deprecation. (#5399)

* Document extended resources and OIR deprecation.

* Updated extended resources doc per reviews.

* reverts extra spacing in _data/tasks.yml

* addresses `kubeadm upgrade` review comments

Feedback from @chenopis, @luxas, and @steveperry-53 addressed with this commit

* HugePages documentation (#5419)

* Update cpu-management-policies.md (#5407)

Fixed the bad link.
Modified "cpu" to "CPU".
Added more 'yaml' as supplement.

* Update RBAC docs for v1 (#5445)

* Add user docs for pod priority and preemption (#5328)

* Add user docs for pod priority and preemption

* Update pod-priority-preemption.md

* More updates

* Update docs/admin/kubeadm.md for 1.8 (#5440)

- Made a couple of minor wording changes (not strictly 1.8 related).
 - Did some reformatting (not strictly 1.8 related).
 - Updated references to the default token TTL (was infinite, now 24 hours).
 - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`.
 - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples.
 - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes.
 - Documented the new `--groups` flag for `kubeadm token create`.
 - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section.
 - Updated the component version table to drop 1.6 and add 1.8.
 - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`.

* Update StatefulSet Basics for 1.8 release (#5398)

* addresses `kubeadm upgrade` review comments

2nd iteration review comments by @luxas

* adds kubelet upgrade section to kubeadm upgrade

* Fix a bulleted list on docs/admin/kubeadm.md. (#5458)

I updated this doc yesterday and I was absolutely sure I fixed this, but I just saw that this commit got lost somehow.

This was introduced recently in #5440.

* Clarify the API to check for device plugins

* Moving Flexvolume to separate out-of-tree section

* addresses `kubeadm upgrade` review comments

CC: @luxas

* fixes kubeadm upgrade index

* Update Stackdriver Logging documentation (#5495)

* Re-update WordPress and MySQL PV doc to use apps/v1beta2 APIs (#5526)

* Update statefulset concepts doc to use apps/v1beta2 APIs (#5420)

* add document on kubectl's behavior regarding initializers (#5505)

* Update docs/admin/kubeadm.md to cover self-hosting in 1.8. (#5497)

This is a new beta feature in 1.8.

* Update kubectl patch doc to use apps/v1beta2 APIs (#5422)

* [1.8] Update "Run Applications" tasks to apps/v1beta2. (#5525)

* Update replicated stateful application task for 1.8.

* Update single instance stateful app task for 1.8.

* Update stateless app task for 1.8.

* Update kubectl patch task for 1.8.

* fix the link of persistent storage (#5515)

* update the admission-controllers.md index.md what-is-kubernetes.md link

* fix the link of persistent storage

* Add quota support for local ephemeral storage (#5493)

* Add quota support for local ephemeral storage

update the doc to this alpha feature

* Update resource-quotas.md

* Updated Deployments concepts doc (#5491)

* Updated Deployments concepts doc

* Addressed comments

* Addressed more comments

* Modify allocatable storage to ephemeral-storage (#5490)

Update the doc to use ephemeral-storage instead of storage

* Revamped concepts doc for ReplicaSet (#5463)

* Revamped concepts doc for ReplicaSet

* Minor changes to call out specific versions for selector defaulting and
immutability

* Addressed doc review comments

* Remove petset documentations (#5395)

* Update docs to use batch/v1beta1 cronjobs (#5475)

* add federation job doc (#5485)

* add federation job doc

* Update job.md

Edits for clarity and consistency

* Update job.md

Fixed a typo

* update DaemonSet concept for 1.8 release (#5397)

* update DaemonSet concept for 1.8 release

* Update daemonset.md

Fix typo. than -> then

* Update bootstrap tokens doc for 1.8. (#5479)

* Update bootstrap tokens doc for 1.8.

This has some changes I missed when I was updating the main kubeadm documention:
 - Bootstrap tokens are now beta, not alpha (kubernetes/enhancements#130)
 - The apiserver flag to enable the authenticator changedin 1.8 (kubernetes/kubernetes#51198)
 - Added `auth-extra-groups` documentaion (kubernetes/kubernetes#50933)
 - Updated the _Token Management with `kubeadm`_ section to link to the main kubeadm docs, since it was just duplicated information.

* Update bootstrap-tokens.md

* Updated the Cassandra tutorial to use apps/v1beta2 (#5548)

* add docs for AllowPrivilegeEscalation (#5448)

Signed-off-by: Jess Frazelle <acidburn@microsoft.com>

* Add local ephemeral storage alpha feature in managing compute resource (#5522)

* Add local ephemeral storage alpha feature in managing compute resource

Since 1.8, we add the local ephemeral storage alpha feature as one
resource type to manage. Add this feature into the doc.

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Added documentation for Metrics Server (#5560)

* authorization: improve authorization debugging docs (#5549)

* Document mount propagation (#5544)

* Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524)

This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

* Add task doc for alpha dynamic kubelet configuration (#5523)

* Fix input/output of selfsubjectaccess review (#5593)

* Add docs for implementing resize (#5528)

* Add docs for implementing resize

* Update admission-controllers.md

* Added link to PVC section

* minor typo fixes

* Update NetworkPolicy concept guide with egress and CIDR changes (#5529)

* update zookeeper tutorial for 1.8 release

* add doc for hostpath type (#5503)

* Federated Hpa feature doc (#5487)

* Federated Hpa feature doc

* Federated Hpa feature doc review fixes

* Update hpa.md

* Update hpa.md

* update cloud controller manager docs for v1.8

* Update cronjob with defaults information (#5556)

* Kubernetes 1.8 reference docs (#5632)

* Kubernetes 1.8 reference docs

* Kubectl reference docs for 1.8

* Update side bar with 1.8 kubectl and api ref docs links

* remove petset.md

* update on state of HostAlias in 1.8 with hostNetwork Pod support (#5644)

* Fix cron job deletion section (#5655)

* update imported docs (#5656)

* Add documentation for certificate rotation. (#5639)

* Link to using kubeadm page

* fix the command output

fix the command output

* fix typo in api/resources reference: "Worloads"

* Add documentation for certificate rotation.

* Create TOC entry for cloud controller manager. (#5662)

* Updates for new versions of API types

* Followup 5655: fix link to garbage collection (#5666)

* Temporarily redirect resources-reference to api-reference. (#5668)

* Update config for 1.8 release. (#5661)

* Update config for 1.8 release.

* Address reviewer comments.

* Switch references in HPA docs from alpha to beta (#5671)

The HPA docs still referenced the alpha version.  This switches them to
talk about v2beta1, which is the appropriate version for Kubernetes 1.8

* Deprecate openstack heat (#5670)

* Fix typo in pod preset conflict example

Move container port definition to the correct line.

* Highlight openstack-heat provider deprecation

The openstack-heat provider for kube-up is being deprecated and will be
removed in a future release.

* Temporarily fix broken links by redirecting. (#5672)

* Fix broken links. (#5675)

* Fix render of code block (#5674)

* Fix broken links. (#5677)

* Add a small note about auto-bootstrapped CSR ClusterRoles (#5660)

* Update kubeadm install doc for v1.8 (#5676)

* add draft workloads api content for 1.8 (#5650)

* add draft workloads api content for 1.8

* edits per review, add tables, for 1.8 workloads api doc

* fix typo

* Minor fixes to kubeadm 1.8 upgrade guide. (#5678)

- The kubelet upgrade instructions should be done on every host, not
  just worker nodes.
- We should just upgrade all packages, instead of calling out kubelet
  specifically. This will also upgrade kubectl, kubeadm, and
  kubernetes-cni, if installed.
- Draining nodes should also ignore daemonsets, and master errors can be
  ignored.
- Make sure that the new kubeadm download is chmoded correctly.
- Add a step to run `kubeadm version` to verify after downloading.
- Manually approve new kubelet CSRs if rotation is enabled (known issue).

* Release 1.8 (#5680)

* Fix versions for 1.8 API ref docs

* Updates for 1.8 kubectl reference docs

* Kubeadm /docs/admin/kubeadm.md cleanup, editing. (#5681)

* Update docs/admin/kubeadm.md (mostly 1.8 related).

This is Fabrizio's work, which I'm committing along with my edits (in a commit on top of this).

* A few of my own edits to clarify and clean up some Markdown.

alban added a commit to kinvolk/kube-spawn that referenced this pull request Nov 26, 2017

kubeadm join: fix TLS on Kubernetes >=1.9
kubeadm got a new option in version 1.8:
--discovery-token-unsafe-skip-ca-verification
See: kubernetes/kubernetes#49520

Since version 1.9, it became mandatory to either use it to skip
verification, or to use --discovery-token-ca-cert-hash=...
See: kubernetes/kubernetes#55468

Since kube-spawn is a developer tool used on one physical machine, use
the former.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.