Enable privileged containers for apiserver and controller #57561

dims · 2017-12-22T17:25:19Z

What this PR does / why we need it:

In OpenStack environment, when there is no metadata service, we
look at the config drive to figure out the metadata. Since we need
to run commands like blkid, we need to ensure that api server and
kube controller are running in the privileged mode.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #47392
Fixes kubernetes/kubeadm#588

Special notes for your reviewer:

Release note:

Fix issue when using OpenStack config drive for node metadata

dims · 2017-12-22T17:26:18Z

/assign @luxas

dims · 2017-12-22T20:57:54Z

/retest

dims · 2017-12-22T22:17:22Z

/open

dims · 2017-12-22T22:17:57Z

/reopen

k8s-ci-robot · 2017-12-22T22:17:58Z

@dims: failed to re-open PR: state cannot be changed. There are no new commits on the dims:enable-privileged-container-for-apiserver-and-controller branch.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dims · 2017-12-22T22:20:08Z

/reopen

kad · 2017-12-22T22:28:41Z

Two questions: 1. unit tests. 2. can it be done with allowing few needed caps instead of fully privileged ?

dims · 2017-12-22T22:38:40Z

@kad we need CAP_SYS_ADMIN as we are mounting/unmounting file systems (similar to http://ceph.com/planet/no-more-privileged-containers-for-ceph-osds/). Any suggestions on the best way to do this?

Thanks for the quick review!

kad · 2017-12-22T22:56:33Z

In case of mounts, probably no suggestions from me at the moment. If CAP_SYS_ADMIN is enough, maybe that would be the best way.

dims · 2017-12-23T00:25:50Z

@kad so leave the current implementation as-is?

kad · 2017-12-27T13:17:56Z

@dims test case still will be good. so if someone will do refactoring on that code later, we will not loose it.
@liggitt what do you think about security/privileged pod for api manager and controller manager for that usage scenario ?

dims · 2018-01-10T20:52:52Z

cc @kubernetes/sig-cluster-lifecycle-pr-reviews

jamiehannaford

I'd also like if we could reduce the permission scope if possible. If it's only possible with privileged, fine with me

jamiehannaford · 2018-01-12T00:58:41Z

cmd/kubeadm/app/phases/controlplane/manifests.go

@@ -104,6 +105,15 @@ func GetStaticPodSpecs(cfg *kubeadmapi.MasterConfiguration, k8sVersion *version.
 		}, mounts.GetVolumes(kubeadmconstants.KubeScheduler)),
 	}

+	if cfg.CloudProvider == "openstack" {


might be worth making openstack a constant and also referencing different providers in the unit tests

@jamiehannaford i see "aws" and "gce" as well. i'll do it in another PR after this one merges

I'd rather this be more generic enablement, b/c this isn't openstack specific. This is a specific os-deployment issue.

Should i remove the if condition for openstack?

Do AWS and GCE also use the config drive for node metadata?

I don't know for sure @jamiehannaford

@dims I'd prefer an opt-in knob to enable privileged (default false), b/c again, this is very specific and not necessarily the common case.

Agree with a specific opt-in knob. I'd consider it acceptable to have documentation for that knob state that often times cloud providers need it to be set and name openstack specifically.

timothysc · 2018-01-17T02:45:29Z

cmd/kubeadm/app/phases/controlplane/manifests.go

+			Privileged: utilpointer.BoolPtr(true),
+		}
+		staticPodSpecs[kubeadmconstants.KubeControllerManager].Spec.Containers[0].SecurityContext = &v1.SecurityContext{
+			Privileged: utilpointer.BoolPtr(true),


Why does the controller manager need to talk with the meta-data service?

when the cloud provider is looking for "which node am i" info, in this specific scenario there is no openstack metadata service and the code looks in the config drive, to be able to mount the config drive it needs privs. we have a search-order (https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/) but in this case there is simply no metadata service and the only choice is the config service where it fails because the kube controller manager does not have enough super powers

It would be nice to put this down into a code comment.

@kargakis ack, if this looks ok otherwise, will happily do that in a little bit

Once the cloud-controller code is excised from the controller manager, is there any reason that we would still need this?

@timothysc correct, when we excise code that calls out to openstack cloud provider from these 2 containers we can yank the privilege flag.

When is the ETA on that excise? 1.10, 1.11? "Hand-wavy" future?

Also, would #58080 work for this use case?

@timothysc we con't currently expose --cloud-provider from kubeadm command line, so that won't help. i've updated the PR with a new field in MasterConfiguration and updated the unit test. hopefully that combination will work for everyone.

@timothysc For 1.10, i do not expect to move at least the openstack cloud provider out of tree. at best we will have an external openstack controller manager that will work for some scenarios with the in-tree one still being the primary one. 1.11 is a distinct possibility.

timothysc · 2018-01-17T02:49:57Z

cmd/kubeadm/app/phases/controlplane/manifests.go

@@ -104,6 +105,15 @@ func GetStaticPodSpecs(cfg *kubeadmapi.MasterConfiguration, k8sVersion *version.
 		}, mounts.GetVolumes(kubeadmconstants.KubeScheduler)),
 	}

+	if cfg.CloudProvider == "openstack" {


I'd rather this be more generic enablement, b/c this isn't openstack specific. This is a specific os-deployment issue.

dims · 2018-01-18T01:24:52Z

/test pull-kubernetes-unit

dims · 2018-01-18T01:24:57Z

/retest

timothysc

minor nit on extraneous shufflings, then lgtm

timothysc · 2018-01-18T03:08:14Z

cmd/kubeadm/app/apis/kubeadm/types.go

 	NodeName             string
 	AuthorizationModes   []string

+	// Name of the cloud provider
+	CloudProvider string


The CloudProvider strings don't really need to be shuffled around, it makes it look like you're changing it for some reason.

Fixed. thanks!

timothysc

/lgtm
/approve

In OpenStack environment, when there is no metadata service, we look at the config drive to figure out the metadata. Since we need to run commands like blkid, we need to ensure that api server and kube controller are running in the privileged mode. So add a new field in MasterConfiguration for specifying that the api server and controller manager (s) need extra privileges. Added a TODO to remove this code when we fully yank out cloud provider specific calls from these processes.

dims · 2018-01-18T15:40:03Z

@timothysc apologies, looks like i left one test broken, fixed it now. will ping when everything goes green.

timothysc · 2018-01-18T15:42:09Z

/lgtm

k8s-ci-robot · 2018-01-18T15:42:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, timothysc

Associated issue: #47392

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kubeadm/OWNERS~~ [timothysc]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2018-01-18T17:36:12Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-01-18T18:27:01Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

bputt · 2018-01-19T16:41:48Z

Will this be in a 1.9.x release?

errordeveloper · 2018-01-22T13:09:45Z

@bputt I think this will land in 1.10 and won't get cherry-picked into 1.9.

dims · 2018-01-22T15:00:22Z

@errordeveloper @bputt Ilya is correct, this will fall into the new feature category (not bug fix)

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 22, 2017

k8s-github-robot assigned mikedanese and kad Dec 22, 2017

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 22, 2017

k8s-ci-robot assigned luxas Dec 22, 2017

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch from c2bb2f2 to db028f9 Compare December 22, 2017 19:24

dims closed this Dec 22, 2017

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch from db028f9 to 475d5c3 Compare December 22, 2017 22:16

k8s-ci-robot reopened this Dec 22, 2017

mikedanese removed their assignment Jan 9, 2018

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Jan 10, 2018

jamiehannaford reviewed Jan 12, 2018

View reviewed changes

timothysc reviewed Jan 17, 2018

View reviewed changes

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch 2 times, most recently from da66c8f to 6baf7c4 Compare January 18, 2018 01:14

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch from 6baf7c4 to 042a8c4 Compare January 18, 2018 02:47

timothysc suggested changes Jan 18, 2018

View reviewed changes

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch from 042a8c4 to 0b421fd Compare January 18, 2018 03:24

k8s-ci-robot assigned timothysc Jan 18, 2018

timothysc approved these changes Jan 18, 2018

View reviewed changes

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 18, 2018

dims added 2 commits January 18, 2018 10:37

Update generated code

658a27c

dims force-pushed the enable-privileged-container-for-apiserver-and-controller branch from 0b421fd to 658a27c Compare January 18, 2018 15:38

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 18, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 18, 2018

k8s-github-robot merged commit afd01c0 into kubernetes:master Jan 18, 2018

Enable privileged containers for apiserver and controller #57561

Enable privileged containers for apiserver and controller #57561

Conversation

dims commented Dec 22, 2017 • edited

dims commented Dec 22, 2017

dims commented Dec 22, 2017

dims commented Dec 22, 2017

dims commented Dec 22, 2017

k8s-ci-robot commented Dec 22, 2017

dims commented Dec 22, 2017

kad commented Dec 22, 2017

dims commented Dec 22, 2017

kad commented Dec 22, 2017

dims commented Dec 23, 2017

kad commented Dec 27, 2017

dims commented Jan 10, 2018

jamiehannaford left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Jan 18, 2018

dims commented Jan 18, 2018

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc left a comment

Choose a reason for hiding this comment

dims commented Jan 18, 2018

timothysc commented Jan 18, 2018

k8s-ci-robot commented Jan 18, 2018

k8s-github-robot commented Jan 18, 2018

k8s-github-robot commented Jan 18, 2018

bputt commented Jan 19, 2018

errordeveloper commented Jan 22, 2018

dims commented Jan 22, 2018

dims commented Dec 22, 2017 •

edited