Experimental support for running kubelet in container #6936

pmorie · 2015-04-16T21:56:41Z

WIP for #6848

googlebot · 2015-04-16T21:56:44Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

pmorie · 2015-04-16T21:59:06Z

So far this PR:

Injects mounters into volume plugins
Adds a new mounter that knows how to nsenter the host's mount namespace from a container
Runs a containerized flag to the kubelet
Turns off kubelet joining cgroup to use devices (will be handled with docker --privileged option) when kubelet is containerized
Injects an nsentering mounter into plugins when kubelet is containerized

It builds; tests are going to be broken.

Next up: make an image that has this kubelet binary and nsenter and test.

pmorie · 2015-04-16T22:19:41Z

@vmarmol @thockin @erictune @eparis @rootfs @vishh

pmorie · 2015-04-17T00:28:54Z

How could I forget @yifan-gu @dchen1107

vishh · 2015-04-17T21:05:02Z

This hack LGTM.

pmorie · 2015-04-17T21:23:58Z

@vishh Yep, this is 99% hack for POC purposes. The mounter injection would be the 1% that we can use.

pmorie · 2015-04-17T21:55:12Z

Now with more hacks. Tmpfs e2e runs locally with kubelet running in container; secrets doesn't. Next step will be to investigate secrets.

pmorie · 2015-04-20T22:09:28Z

Code in this branch works with docker 1.6. Currently, there's a dependency on 1.6 because it switches the mount propogation mode of bind mounts from MOUNT_PRIVATE to MOUNT_SLAVE, so mounts made to host fs volumes (made from the host's mount ns) bind-mounted into containers will be propagated to the bind-mounts. Docker 1.5 uses MOUNT_PRIVATE. I've tested by doing an e2e run locally. There are some cases that don't work, but they appear to be the usual suspects and most of them shouldn't be run anyway against local (will open PRs up to skip these).

One nit that I found: mounts for volumes are not currently cleaned up when the kubelet is run in a container (ie, umount fails with device or resource busy). Those mount points may show as busy because they're under the bind-mount of the kubelet root dir. Need to do more digging on this to follow up.

With that said, I'm curious about how folks feel about some of this code going in so that we can experiment further without carrying the patch. I would want to:

Rebase on top of Implement Mount interface using mount(8) and umount(8) #6400 when it goes in
Pull the mounter to use up to being a field of the kubelet that the creator of the kubelet can supply (instead of having the getMounter method that would introduce a dependency on `NsenterMounter)
Pull the NsenterMounter out for now or use some other special casing around it which is clearly marked as experimental

Unit and integration tests still broken; will fix those up next.

Thanks to @eparis and @vbatts again for support on getting this working.

Any thoughts @thockin @vishk @smarterclayton?

eparis · 2015-04-20T22:17:35Z

cmd/kubelet/app/server.go

@@ -204,6 +205,9 @@ func (s *KubeletServer) AddFlags(fs *pflag.FlagSet) {
 	// Flags intended for testing, not recommended used in production environments.
 	fs.BoolVar(&s.ReallyCrashForTesting, "really_crash_for_testing", s.ReallyCrashForTesting, "If true, when panics occur crash. Intended for testing.")
 	fs.Float64Var(&s.ChaosChance, "chaos_chance", s.ChaosChance, "If > 0.0, introduce random client errors and latency. Intended for testing. [default=0.0]")
+
+	// HACK: are you containerized?
+	fs.BoolVar(&s.Containerized, "containerized", s.Containerized, "Indicates whether kubelet is running in a container")


Would it be better to check for /.dockerenv or /.dockerinit instead of having to make the user remember on the command line?

@eparis Eventually that seems sane.

pmorie · 2015-04-27T22:16:54Z

Rebased, will test later. Next up for this is fixing tests. Still waiting on #6400.

pmorie · 2015-04-27T22:18:56Z

I think I've got a handle on getting unmount to work correctly; I think I need to perform the unmount first in the kubelet container's mount ns, and then on the host.

pmorie · 2015-04-29T22:00:23Z

Sweet, more rebasing later.

pmorie · 2015-05-01T01:01:49Z

I ran emptyDir and secrets E2Es locally with SELinux enforcing against containerized kubelet, both passed.

pmorie · 2015-05-01T01:50:30Z

@vmarmol My rebase party is complete, I think this is ready for final review.

pmorie · 2015-05-01T20:17:22Z

I'm blocked at the moment from doing the E2E build because of the bug this commit fixes:

rhatdan/moby1@350a636

They're cutting a new 1.6 package for fedora but it will probably be a couple days before it shows up in repos. :-/

pmorie · 2015-05-01T20:18:00Z

@vmarmol If you are running docker 1.5 locally and could find it in your heart do kick off an e2e run, I would definitely owe you 🍻

vmarmol · 2015-05-01T22:02:39Z

Spoke to @pmorie on IRC and we're gonna split the PR into 4:

Injecting mounter
nsenter mounter
Local Dockerized Kubelet
Building Dockerized Kubelet

vmarmol · 2015-05-01T22:04:06Z

@pmorie the default Docker on GCE is still 1.5 so I'd be happy to test :D I'll kickoff an e2e with this branch

pmorie · 2015-05-04T20:18:53Z

Closing this out since we're splitting out separate PRs.

googlebot added the cla: no label Apr 16, 2015

bgrant0607 assigned dchen1107 Apr 17, 2015

pmorie force-pushed the run-in-container branch from c7f857c to 32256b0 Compare April 17, 2015 21:54

pmorie force-pushed the run-in-container branch 2 times, most recently from 2c6986c to 46db2ef Compare April 20, 2015 21:35

eparis reviewed Apr 20, 2015
View reviewed changes

pmorie force-pushed the run-in-container branch 3 times, most recently from 9dd48a2 to 31f9f88 Compare April 27, 2015 22:16

pmorie force-pushed the run-in-container branch 9 times, most recently from b4a01cb to b0eef85 Compare April 29, 2015 21:45

pmorie force-pushed the run-in-container branch from b0eef85 to e09b7d3 Compare April 29, 2015 22:07

pmorie force-pushed the run-in-container branch from 339e429 to 8f5eae4 Compare May 1, 2015 01:01

pmorie force-pushed the run-in-container branch 5 times, most recently from dff50a2 to 7c88a6b Compare May 1, 2015 01:49

pmorie changed the title ~~WIP: Run kubelet in container~~ Experimental support for running kubelet in container May 1, 2015

pmorie force-pushed the run-in-container branch 5 times, most recently from 449895d to 3d25e40 Compare May 1, 2015 19:51

pmorie force-pushed the run-in-container branch 3 times, most recently from c79c936 to f45cbb7 Compare May 4, 2015 14:09

Experimental support for running kubelet in container

0d64418

pmorie force-pushed the run-in-container branch from f45cbb7 to 0d64418 Compare May 4, 2015 14:11

This was referenced May 4, 2015

Inject mounter into volume plugins #7702

Merged

Add NsenterMounter mount implementation #7703

Merged

pmorie closed this May 4, 2015

errordeveloper mentioned this pull request Jan 19, 2016

Local docker cluster doesn't populate service account token #11984

Closed

This was referenced Mar 19, 2019

Dockerfiles and scripts to containerize edge controller kubeedge/kubeedge#246

Merged

Dockerfiles and scripts to containerize edge core kubeedge/kubeedge#247

Merged

Experimental support for running kubelet in container #6936

Experimental support for running kubelet in container #6936

pmorie commented Apr 16, 2015

googlebot commented Apr 16, 2015

pmorie commented Apr 16, 2015

pmorie commented Apr 16, 2015

pmorie commented Apr 17, 2015

vishh commented Apr 17, 2015

pmorie commented Apr 17, 2015

pmorie commented Apr 17, 2015

pmorie commented Apr 20, 2015

eparis Apr 20, 2015

pmorie Apr 22, 2015

pmorie commented Apr 27, 2015

pmorie commented Apr 27, 2015

pmorie commented Apr 29, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

vmarmol commented May 1, 2015

vmarmol commented May 1, 2015

pmorie commented May 4, 2015

Experimental support for running kubelet in container #6936

Experimental support for running kubelet in container #6936

Conversation

pmorie commented Apr 16, 2015

googlebot commented Apr 16, 2015

pmorie commented Apr 16, 2015

pmorie commented Apr 16, 2015

pmorie commented Apr 17, 2015

vishh commented Apr 17, 2015

pmorie commented Apr 17, 2015

pmorie commented Apr 17, 2015

pmorie commented Apr 20, 2015

eparis Apr 20, 2015

Choose a reason for hiding this comment

pmorie Apr 22, 2015

Choose a reason for hiding this comment

pmorie commented Apr 27, 2015

pmorie commented Apr 27, 2015

pmorie commented Apr 29, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

pmorie commented May 1, 2015

vmarmol commented May 1, 2015

vmarmol commented May 1, 2015

pmorie commented May 4, 2015