Add e2e regression tests for the kubelet being secure #64140

dixudx · 2018-05-22T09:05:20Z

What this PR does / why we need it:
This PR does,

The kubelet cAdvisor port (4194) can't be reached, neither via the API server proxy nor directly on the public IP address
The kubelet read-only port (10255) can't be reached, neither via the API server proxy nor directly on the public IP address
The kubelet can delegate ServiceAccount tokens to the API server
The kubelet's main port (10250) has both authentication (should fail with no credentials) and authorization (should fail with insufficient permissions) set-up

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/kubeadm#838

Special notes for your reviewer:
/cc luxas tallclair
Release note:

Add e2e regression tests for the kubelet being secure

dixudx · 2018-05-22T09:06:59Z

/cc @kubernetes/sig-node-proposals @kubernetes/sig-auth-proposals

luxas

This looks great overall.
Thanks for this valuable contribution @dixudx 👍!
I left a couple of initial comments.

I'd love a review from @kubernetes/sig-auth-pr-reviews, and a comment from @kubernetes/sig-architecture-pr-reviews regarding eventually graduating It("Should make the kubelet's main port 10250 enforce authentication for client requests") a Conformance test as @tallclair commented in kubernetes/kubeadm#838 (comment)

test/e2e/auth/node_authn.go

luxas · 2018-05-22T17:20:47Z

test/e2e/auth/node_authn.go

+		Expect(err).NotTo(HaveOccurred())
+		Expect(len(nodeList.Items)).NotTo(Equal(0))
+		nodeName = nodeList.Items[0].Name
+		sa, err := f.ClientSet.CoreV1().ServiceAccounts(ns).Get("default", metav1.GetOptions{})


Add a comment why you do this. (I think it's to make sure the ServiceAccount admission controller is enabled etc., so secret generation on SA creation works, right?)

Correct. Add a comment is better.

test/e2e/auth/node_authn.go

luxas · 2018-05-22T17:27:58Z

test/e2e/auth/node_authz.go

@@ -179,4 +181,18 @@ var _ = SIGDescribe("[Feature:NodeAuthorizer]", func() {
 		err := c.CoreV1().Nodes().Delete("foo", &metav1.DeleteOptions{})
 		Expect(apierrors.IsForbidden(err)).Should(Equal(true))
 	})
+
+	framework.ConformanceIt("The kubelet's main port 10250 should fail with the Forbidden error", func() {


Don't add conformance here please. You can add a comment though to maybe do it in the future though.
cc @tallclair

test/e2e/lifecycle/kubelet_security.go

luxas · 2018-05-22T17:30:37Z

test/e2e/lifecycle/kubelet_security.go

+
+	prefix := "/api/v1"
+	// make sure kubelet readonly (10255) and cadvisor (4194) ports are disabled via API server proxy
+	It("should not be able to contact the readonly port 10255", func() { portProxyDisabledTest(f, prefix+"/nodes/", ":10255/pods/") })


suggest: should not be able to proxy to the readonly kubelet port 10255 using proxy subresource to be consistent with what you have below

test/e2e/lifecycle/kubelet_security.go

luxas

Thanks a lot @dixudx!!
I'm fine for the moment, the high level functionality is there, deferring the rest of the review to SIG Auth reviewers.
I'm gonna try running these locally in a minute to make sure everything actually works 😄

/approve
/lgtm

test/e2e/auth/node_authn.go

test/e2e/auth/node_authz.go

liggitt · 2018-05-23T19:38:04Z

while I like the idea of exercising the described tests, these e2e tests don't actually do that. the only reason CI is green with them added is because they're behind feature tags that don't actually run them (or the API server happens to return the same response we were hoping to see from the kubelet)

tallclair · 2018-05-23T21:05:03Z

I recommend proxying the tests through shell commands run in a container and pointed at the kubelets cluster-internal IP address. See the apparmor test for an example of how you might do this:
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/apparmor.go#L60

Automatic merge from submit-queue (batch tested with PRs 64174, 64187, 64216, 63265, 64223). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubeadm: Improve the kubelet default configuration security-wise **What this PR does / why we need it**: - Disables the readonly port for the kubelets in the cluster - Enables delegated SA token authentication for the secure kubelet port (GCE also did this ref: #58178) - Follows up #63912 to move the last flag from the system dropin to the ComponentConfig **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes kubernetes/kubeadm#732 Fixes kubernetes/kubeadm#650 Replaces #57997 **Special notes for your reviewer**: In order to make sure this actually works, or that clusters actually are secure, we're adding e2e tests for this: kubernetes/kubeadm#838 & #64140 Depends on #63912 **Release note**: ```release-note [action required] kubeadm: kubelets in kubeadm clusters now disable the readonly port (10255). If you're relying on unauthenticated access to the readonly port, please switch to using the secure port (10250). Instead, you can now use ServiceAccount tokens when talking to the secure port, which will make it easier to get access to e.g. the `/metrics` endpoint of the kubelet securely. ``` @kubernetes/sig-cluster-lifecycle-pr-reviews @kubernetes/sig-auth-pr-reviews FYI

k8s-ci-robot · 2018-05-29T07:29:04Z

New changes are detected. LGTM label has been removed.

tallclair · 2018-06-01T22:02:53Z

Yeah, this isn't a conformance test, so I don't think it needs to be tied to a version. We can merge it when it's ready.

dixudx · 2018-06-02T05:36:47Z

@jberkus @tallclair Addressed the comments. PTAL. Thanks.

tallclair · 2018-06-04T17:23:48Z

test/e2e/auth/node_authn.go

+		Expect(len(nodeList.Items)).NotTo(BeZero())
+
+		pickedNode := nodeList.Items[0]
+		// Internal IPs are meaningless from the e2e test


This comment only applies to the test runner. Since you're execing both the tests through a pod, you can test the internal IPs here.

tallclair · 2018-06-04T17:24:23Z

test/e2e/auth/node_authn.go

+		Expect(len(sa.Secrets)).NotTo(BeZero())
+	})
+
+	It("The kubelet's main port 10250 should fail with no credentials", func() {


nit: s/fail/reject requests/

tallclair · 2018-06-04T17:28:06Z

test/e2e/auth/node_authn.go

+					"--insecure",
+					fmt.Sprintf("https://%s:%v/metrics", nodeIP, ports.KubeletPort),
+				},
+				"401",


I'd prefer to check for 401 OR 403, that way you're not depending on a specific implementation as much.

tallclair · 2018-06-04T17:30:57Z

test/e2e/auth/node_authn.go

+					"-s",
+					"-I",
+					"--insecure",
+					fmt.Sprintf(`--header "Authorization: Bearer %s"`, "`cat /var/run/secrets/kubernetes.io/serviceaccount/token`"),


curl doesn't know how to interpret backticks - you need to run this command in a shell. I.e.:

[]string{"sh", "-c", "curl -sI --insecure --header "Authorization: Bearer `cat /var/run/secrets/kubernetes.io/serviceaccount/token`" + url}

tallclair · 2018-06-04T17:31:09Z

test/e2e/auth/node_authn.go

+					fmt.Sprintf(`--header "Authorization: Bearer %s"`, "`cat /var/run/secrets/kubernetes.io/serviceaccount/token`"),
+					fmt.Sprintf("https://%s:%v/metrics", nodeIP, ports.KubeletPort),
+				},
+				"403",


tallclair · 2018-06-04T17:34:39Z

test/e2e/lifecycle/kubelet_security.go

+		Expect(statusCode).NotTo(Equal(http.StatusOK))
+	})
+	It("should not be able to proxy to cadvisor port 4194 using proxy subresource", func() {
+		result, err := framework.NodeProxyRequest(f.ClientSet, nodeName, "proxy/containers/", 4194)


Are you sure the path should include proxy here?

tallclair · 2018-06-04T17:42:58Z

There seems to be some confusion about what internal IPs and external IPs are.

External IPs are available to the entire internet.
Internal IPs are from a private address space, meaning they are only meaningful within the private network. In our case, that means running in the cluster.

The E2E test runner runs outside the cluster being tested, which means that it can only see the external IP addresses (actually, it can see the internal IP addresses, but they are resolved within the test runner's private network, so they probably resolve to different devices, if at all). When you run a pod, it runs in the cluster, meaning the pod can see the internal addresses, this is why an tests that look at the internal IPs should be run through a pod.

Hopefully that clears it up?

tallclair · 2018-06-05T22:55:01Z

test/e2e/auth/node_authn.go

+		pod := createNodeAuthTestPod(f)
+		for _, nodeIP := range nodeIPs {
+			// Anonymous authentication is disabled by default
+			result, err := framework.LookForStringInPodExec(pod.Namespace,


No reason to use this method if you're not looking for the string. How about just:

result := framework.RunHostCmdOrDie(ns, pod.Name, fmt.Sprintf("curl -sIk -o /dev/null -w '%{http_code}' https://%s:%v/metrics", nodeIP, ports.KubeletPort)) Expect(result).To(Or(Equal("401"), Equal("403")), "the kubelet's main port 10250 should reject requests with no credentials")

tallclair · 2018-06-05T23:03:59Z

test/e2e/auth/node_authn.go

+		pod := createNodeAuthTestPod(f)
+
+		for _, nodeIP := range nodeIPs {
+			result, err := framework.LookForStringInPodExec(pod.Namespace,


Same as above comment.

tallclair · 2018-06-05T23:05:48Z

test/e2e/auth/node_authn.go

+				time.Minute)
+			Expect(err).NotTo(HaveOccurred())
+
+			if !strings.Contains(result, "403") && !strings.Contains(result, "401") {


Expect(result).To(And(BeNumerically(">=", 200), BeNumerically("<", 300)), ...)

I don't particularly care for the gomega DSL, so feel free to do these checks explicitly, but semantically this is the condition you should check for.

You might be able to simplify this to just: Expect(result).To(Equal("200"), ...)

Here we're using a new SA token when sending requests to the kubelet. The expected result should be 403, and authentication is okay.

Expect(result).To(Equal("200"), ...)

@tallclair I don't quite understand where 200 comes.

tallclair · 2018-06-05T23:07:24Z

test/e2e/auth/node_authn.go

+
+		pickedNode := nodeList.Items[0]
+		// Here we only need to care about Internal IP, since the pods running in the cluster
+		// can see the internal addresses.


This comment still doesn't make sense to me. Why did you drop the external IPs?

So we add external IPs back again? I was testing both internal IPs and external IPs before.

tallclair · 2018-06-05T23:10:04Z

test/e2e/lifecycle/kubelet_security.go

+	BeforeEach(func() {
+		nodes := framework.GetReadySchedulableNodesOrDie(f.ClientSet)
+		Expect(len(nodes.Items)).NotTo(BeZero())
+		node = &nodes.Items[0]


The 0th node is usually the master, which might be configured differently. Still a good idea to check it though, but maybe pick a few (or all of them)?

Actually all the nodes here are schedulable and ready. Usually the master node is tainted, which will be filtered due to unschedulable.

So the 0th node is usually not the master. And 0th node is used widely in our e2e test scenarios.

kubernetes/test/e2e/node/mount_propagation.go

Lines 86 to 89 in ad9722c

// Pick a node where all pods will run.

nodes := framework.GetReadySchedulableNodesOrDie(f.ClientSet)

Expect(len(nodes.Items)).NotTo(BeZero(), "No available nodes for scheduling")

node := &nodes.Items[0]

liggitt · 2018-06-06T14:55:51Z

per #64140 (comment) this isn't tied to v1.11 or blocking the release

/milestone clear

test/e2e/auth/node_authn.go

dixudx · 2018-06-20T01:25:53Z

@luxas @tallclair Needs your lgtm. Thanks.

tallclair · 2018-06-21T00:12:12Z

/lgtm

k8s-ci-robot · 2018-06-21T00:12:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dixudx, luxas, tallclair, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dixudx · 2018-06-21T01:39:58Z

@tallclair @luxas Seems the bot does not want to lgtm.

k8s-github-robot · 2018-06-21T09:23:31Z

/test all

Tests are more than 96 hours old. Re-running tests.

k8s-github-robot · 2018-06-21T11:15:56Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-ci-robot · 2018-06-21T12:17:30Z

@dixudx: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-bazel-test	`924df8a`	link	`/test pull-kubernetes-bazel-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-github-robot · 2018-06-21T12:40:05Z

Automatic merge from submit-queue (batch tested with PRs 64140, 64898, 65022, 65037, 65027). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot requested a review from luxas May 22, 2018 09:05

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 22, 2018

k8s-ci-robot requested a review from tallclair May 22, 2018 09:05

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 22, 2018

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. kind/design Categorizes issue or PR as related to design. sig/auth Categorizes an issue or PR as relevant to SIG Auth. labels May 22, 2018

luxas reviewed May 22, 2018

View reviewed changes

k8s-ci-robot added the sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. label May 22, 2018

luxas assigned liggitt, luxas and tallclair May 22, 2018

dixudx force-pushed the add_e2e_kubelet_port branch from 5b4cb58 to b640ca7 Compare May 23, 2018 06:05

luxas approved these changes May 23, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

luxas mentioned this pull request May 23, 2018

kubeadm: Improve the kubelet default configuration security-wise #64187

Merged

liggitt reviewed May 23, 2018

View reviewed changes

test/e2e/auth/node_authn.go Outdated Show resolved Hide resolved

liggitt reviewed May 23, 2018

View reviewed changes

test/e2e/auth/node_authn.go Outdated Show resolved Hide resolved

liggitt reviewed May 23, 2018

View reviewed changes

test/e2e/auth/node_authn.go Outdated Show resolved Hide resolved

liggitt reviewed May 23, 2018

View reviewed changes

test/e2e/auth/node_authz.go Outdated Show resolved Hide resolved

dixudx force-pushed the add_e2e_kubelet_port branch from b640ca7 to fe65a18 Compare May 29, 2018 07:29

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 29, 2018

dixudx force-pushed the add_e2e_kubelet_port branch from fe65a18 to bff5c13 Compare May 29, 2018 08:00

tallclair removed this from the v1.11 milestone Jun 1, 2018

dixudx force-pushed the add_e2e_kubelet_port branch from 21f3f54 to c4472c8 Compare June 2, 2018 04:04

tallclair reviewed Jun 4, 2018

View reviewed changes

dixudx force-pushed the add_e2e_kubelet_port branch from c4472c8 to ff92c01 Compare June 5, 2018 08:00

tallclair reviewed Jun 5, 2018

View reviewed changes

dixudx force-pushed the add_e2e_kubelet_port branch 2 times, most recently from 27c98bf to f71ad8c Compare June 6, 2018 03:02

luxas removed the kind/design Categorizes issue or PR as related to design. label Jun 6, 2018

luxas added this to the v1.11 milestone Jun 6, 2018

k8s-ci-robot removed this from the v1.11 milestone Jun 6, 2018

tallclair reviewed Jun 12, 2018

View reviewed changes

test/e2e/auth/node_authn.go Outdated Show resolved Hide resolved

dixudx added 2 commits June 12, 2018 17:05

add e2e regression tests for the kubelet being secure

8094e1c

auto-generated files

924df8a

dixudx force-pushed the add_e2e_kubelet_port branch from f71ad8c to 924df8a Compare June 12, 2018 09:05

luxas mentioned this pull request Jun 12, 2018

Add e2e regression tests for the kubelet being secure kubernetes/kubeadm#838

Closed

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 21, 2018

k8s-github-robot merged commit 0b3af19 into kubernetes:master Jun 21, 2018

dixudx deleted the add_e2e_kubelet_port branch June 21, 2018 12:49

	// Pick a node where all pods will run.
	nodes := framework.GetReadySchedulableNodesOrDie(f.ClientSet)
	Expect(len(nodes.Items)).NotTo(BeZero(), "No available nodes for scheduling")
	node := &nodes.Items[0]

Add e2e regression tests for the kubelet being secure #64140

Add e2e regression tests for the kubelet being secure #64140

Conversation

dixudx commented May 22, 2018

dixudx commented May 22, 2018

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas left a comment

Choose a reason for hiding this comment

liggitt commented May 23, 2018

tallclair commented May 23, 2018

k8s-ci-robot commented May 29, 2018

tallclair commented Jun 1, 2018

dixudx commented Jun 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair commented Jun 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Jun 6, 2018

dixudx commented Jun 20, 2018

tallclair commented Jun 21, 2018

k8s-ci-robot commented Jun 21, 2018

dixudx commented Jun 21, 2018

k8s-github-robot commented Jun 21, 2018

k8s-github-robot commented Jun 21, 2018

k8s-ci-robot commented Jun 21, 2018 • edited Loading

k8s-github-robot commented Jun 21, 2018

k8s-ci-robot commented Jun 21, 2018 •

edited

Loading