Merge pull request #5317 from gambol99/node_registration

Node Authorization Service

Makefile

@@ -665,6 +665,10 @@ bazel-crossbuild-protokube-image:
 bazel-crossbuild-kube-discovery-image:
 	bazel build --experimental_platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //images:kube-discovery.tar
 
+.PHONY: bazel-crossbuild-node-authorizer-image
+bazel-crossbuild-node-authorizer-image:
+	bazel build --experimental_platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //images:node-authorizer.tar
+
 .PHONY: bazel-push
 # Will always push a linux-based build up to the server
 bazel-push: bazel-crossbuild-nodeup
@@ -703,6 +707,12 @@ push-kube-discovery:
 	docker tag bazel/kube-discovery/images:kube-discovery ${DOCKER_REGISTRY}/kube-discovery:${DOCKER_TAG}
 	docker push ${DOCKER_REGISTRY}/kube-discovery:${DOCKER_TAG}
 
+.PHONY: push-node-authorizer
+push-node-authorizer:
+	bazel run //node-authorizer/images:node-authorizer
+	docker tag bazel/node-authorizer/images:node-authorizer ${DOCKER_REGISTRY}/node-authorizer:${DOCKER_TAG}
+	docker push ${DOCKER_REGISTRY}/node-authorizer:${DOCKER_TAG}
+
 .PHONY: bazel-protokube-export
 bazel-protokube-export:
 	mkdir -p ${BAZELIMAGES}

docs/cluster_spec.md

@@ -179,6 +179,22 @@ You could use the [fileAssets](https://github.com/kubernetes/kops/blob/master/do
 
 Example policy file can be found [here](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/audit/audit-policy.yaml)
 
+#### bootstrap tokens
+
+Read more about this here: https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
+
+```yaml
+spec:
+  kubeAPIServer:
+    enableBootstrapTokenAuth: true
+```
+
+By enabling this feature you instructing two things;
+- master nodes will bypass the bootstrap token but they _will_ build kubeconfigs with unique usernames in the system:nodes group _(this ensure's the master nodes confirm with the node authorization mode https://kubernetes.io/docs/reference/access-authn-authz/node/)_
+- secondly the nodes will be configured to use a bootstrap token located by default at `/var/lib/kubelet/bootstrap-kubeconfig` _(though this can be override in the kubelet spec)_. The nodes will sit the until a bootstrap file is created and once available attempt to provision the node.
+
+**Note** enabling bootstrap tokens does not provision bootstrap tokens for the worker nodes. Under this configuration it is assumed a third-party process is provisioning the tokens on behalf of the worker nodes. For the full setup please read [Node Authorizer Service](https://github.com/kubernetes/kops/blob/master/docs/node_authorization.md)
+
 #### Max Requests Inflight
 
 The maximum number of non-mutating requests in flight at a given time. When the server exceeds this, it rejects requests. Zero for no limit. (default 400)

docs/node_authorization.md

@@ -0,0 +1,45 @@
+### **Node Authorization Service**
+
+The [node authorization service] is an experimental service which in the absence of a kops-apiserver provides the distribution of tokens to the worker nodes. Bootstrap tokens provide worker nodes a short-time credential to request access kubeconfig certificate. A gist of the flow is;
+
+- a secret of type `bootstrap.kubernetes.io/token` is created on behalf of a node in the kube-system namespace.
+- the token is distributed to the node by _some_ means and then used as the bearer token of the initial request to the kubernetes api.
+- the token itself is bound to the cluster role which grants permission to generate a CSR, an additional cluster role provides access for the controller to auto-approve this CSR requests as well.
+- two certificates are generated by the kubelet using bootstrap process, one for the kubelet api and the other a client certificate to the kubelet itself.
+- the client certificate by default is added into the system:nodes rbac group _(note, if you are using PSP this is automatically bound by kops on your behalf)_.
+- the kubelet at this point has a server certificate and the client api certificate and good to go.
+
+#### **Integretion with Kops**
+
+The node authorization service is run on the master as a daemonset, by default dns is _node-authorizer-interanl.dns_zone_:10443 and added via same mechanism at the internal kube-apiserver i.e. annotations on the kube-apiserver pods which is picked up the dns-controller and added to the dns zone.
+
+When the node authorization service is enabled a systemd _(node-authorizer.service)_ unit is added on the worker nodes. This runs the node-authorizer in client mode and connects to the authorization service requesting a bootstrap token.
+
+#### **Authorizers**
+
+The node authorizer currently supports two authorizers; aws and alwaysallow. The latter is self-explanatory, as for the aws authorizer, in order for a request to be authorized the following checks are performed.
+
+- the worker node retrieves the [pkcs7 signed instance document](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html) from the metadata service; this is unique for each instance and available only to them.
+- the client connects using a client certificate which is first checked and passes the instance document to the authorization service.
+- the signed instance document is validated against the public certificates from AWS.
+- we check the node exists and is running.
+- we check the node is running in our region.
+- we check the node is running in our vpc.
+- we check the node is tagged with the correct kubernetes tag.
+- we check the ip address of the client requesting the document is the same the instance document.
+- we check that the node has not already registered.
+
+Assuming all the conditions are met a secret token is generated and returned to the client to continue the provising of the worker node.
+
+#### **Enabling the Node Authorization Service**
+
+Enabling the node authorization service is as follows; firstly you must enable the feature flag as node authorization is still experimental; export KOPS_FEATURE_FLAGS=EnableNodeAuthorization
+
+```
+# in the cluster spec
+nodeAuthorization:
+  # enable the service under the node authorization section, please review the settings in the components.go
+  nodeAuthorizer: {}
+```
+
+Note, by default this will also switch on the [Node authorization](https://kubernetes.io/docs/reference/access-authn-authz/node/) and RBAC mode. We would also suggest turning on the NodeRestriction admission controller.

hack/.packages

@@ -29,6 +29,12 @@ k8s.io/kops/dnsprovider/pkg/dnsprovider/rrstype
 k8s.io/kops/dnsprovider/pkg/dnsprovider/tests
 k8s.io/kops/examples/kops-api-example
 k8s.io/kops/kube-discovery/cmd/kube-discovery
+k8s.io/kops/node-authorizer/cmd/node-authorizer
+k8s.io/kops/node-authorizer/pkg/authorizers/alwaysallow
+k8s.io/kops/node-authorizer/pkg/authorizers/aws
+k8s.io/kops/node-authorizer/pkg/client
+k8s.io/kops/node-authorizer/pkg/server
+k8s.io/kops/node-authorizer/pkg/utils
 k8s.io/kops/nodeup/pkg/bootstrap
 k8s.io/kops/nodeup/pkg/distros
 k8s.io/kops/nodeup/pkg/model
@@ -92,6 +98,7 @@ k8s.io/kops/pkg/model/alimodel
 k8s.io/kops/pkg/model/awsmodel
 k8s.io/kops/pkg/model/components
 k8s.io/kops/pkg/model/components/etcdmanager
+k8s.io/kops/pkg/model/components/node-authorizer
 k8s.io/kops/pkg/model/defaults
 k8s.io/kops/pkg/model/domodel
 k8s.io/kops/pkg/model/gcemodel

node-authorizer/.gitignore

@@ -0,0 +1,3 @@
+*.swp
+bin/
+tests/

node-authorizer/Dockerfile

@@ -0,0 +1,23 @@
+# Copyright 2018 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM fedora:27
+LABEL Name=node-authorizer \
+      Release=https://github.com/kubernetes/kops \
+      Url=https://github.com/kubernetes/kops \
+      Help=https://github.com/kubernetes/kops\issues
+
+ADD bin/node-authorizer /node-authorizer
+
+ENTRYPOINT ["/node-authorizer"]

node-authorizer/cmd/node-authorizer/BUILD.bazel

@@ -0,0 +1,25 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "client.go",
+        "main.go",
+        "server.go",
+    ],
+    importpath = "k8s.io/kops/node-authorizer/cmd/node-authorizer",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//node-authorizer/pkg/authorizers/alwaysallow:go_default_library",
+        "//node-authorizer/pkg/authorizers/aws:go_default_library",
+        "//node-authorizer/pkg/client:go_default_library",
+        "//node-authorizer/pkg/server:go_default_library",
+        "//vendor/github.com/urfave/cli:go_default_library",
+    ],
+)
+
+go_binary(
+    name = "node-authorizer",
+    embed = [":go_default_library"],
+    visibility = ["//visibility:public"],
+)

node-authorizer/cmd/node-authorizer/client.go

@@ -0,0 +1,104 @@
+/*
+Copyright 2018 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"time"
+
+	"k8s.io/kops/node-authorizer/pkg/client"
+
+	"github.com/urfave/cli"
+)
+
+// addClientCommand creates and returns a client command
+func addClientCommand() cli.Command {
+	return cli.Command{
+		Name:  "client",
+		Usage: "starts the service in a client mode and attempts to acquire a bootstrap token",
+
+		Flags: []cli.Flag{
+			cli.StringFlag{
+				Name:   "authorizer",
+				Usage:  "provider we should use to authorize the node registration `NAME`",
+				EnvVar: "AUTHORIZER",
+				Value:  "aws",
+			},
+			cli.StringFlag{
+				Name:   "node-url",
+				Usage:  "the url for the node authorizer service `URL`",
+				EnvVar: "NODE_AUTHORIZER_URL",
+			},
+			cli.StringFlag{
+				Name:   "kubeapi-url",
+				Usage:  "the url for the kubernetes api `URL`",
+				EnvVar: "KUBEAPI_URL",
+			},
+			cli.StringFlag{
+				Name:   "kubeconfig",
+				Usage:  "location to write bootstrap token config `PATH`",
+				EnvVar: "KUBECONFIG_BOOTSTRAP",
+				Value:  "/var/run/kubelet/kubelet-bootstrap.yml",
+			},
+			cli.StringFlag{
+				Name:   "tls-client-ca",
+				Usage:  "file containing the certificate authority used to verify node endpoint `PATH`",
+				EnvVar: "TLS_CLIENT_CA",
+			},
+			cli.StringFlag{
+				Name:   "tls-cert",
+				Usage:  "file containing the client certificate `PATH`",
+				EnvVar: "TLS_CERT",
+			},
+			cli.StringFlag{
+				Name:   "tls-private-key",
+				Usage:  "file containing the client private key `PATH`",
+				EnvVar: "TLS_PRIVATE_KEY",
+			},
+			cli.DurationFlag{
+				Name:   "interval",
+				Usage:  "an interval to wait between failed request `DURATION`",
+				EnvVar: "INTERVAL",
+				Value:  3 * time.Second,
+			},
+			cli.DurationFlag{
+				Name:   "timeout",
+				Usage:  "the max time to wait before timing out `DURATION`",
+				EnvVar: "TIMEOUT",
+				Value:  30 * time.Second,
+			},
+		},
+
+		Action: func(ctx *cli.Context) error {
+			return actionClientCommand(ctx)
+		},
+	}
+}
+
+// actionClientCommand is the client action
+func actionClientCommand(ctx *cli.Context) error {
+	return client.New(&client.Config{
+		Authorizer:        ctx.String("authorizer"),
+		Interval:          ctx.Duration("interval"),
+		KubeAPI:           ctx.String("kubeapi-url"),
+		KubeConfigPath:    ctx.String("kubeconfig"),
+		NodeURL:           ctx.String("node-url"),
+		TLSCertPath:       ctx.String("tls-cert"),
+		TLSClientCAPath:   ctx.String("tls-client-ca"),
+		TLSPrivateKeyPath: ctx.String("tls-private-key"),
+		Timeout:           ctx.Duration("timeout"),
+	})
+}