host: Replace libvirt-lxc backend with libcontainer #3030

lmars · 2016-07-01T18:58:25Z

Opening as an early PR to see what happens in CI.

Closes #38
Closes #3021
Closes #2553
Closes #2367
Closes #2725
Closes #2803

lmars · 2016-07-03T01:31:20Z

There are still a couple of test failures, but I think these are CI flakes present on master, so this is now ready for review.

An overview of how containers work with libcontainer:

flynn-host exec's flynn-init with libcontainer-init as a command line argument, and the _LIBCONTAINER_INITPIPE env var pointing at a pipe file descriptor opened by flynn-host
flynn-init receives binary netlink data followed by a JSON configuration payload from the pipe which it uses to create namespaces, setup networking etc. (this is handled transparently by libcontainer)
flynn-init exec's /.containerinit inside the container (which is just a bind mount of flynn-init from the host), and then containerinit proceeds as it does currently

Some notes:

we continue to use cgroup partitions as we did with libvirt, but these are now named /flynn/{system,user,background} (rather than libvirt's /machine/{system,user,background}.partition)
starting a container no longer leaves a supervisor process in the root mount namespace, so we no longer need nsumount and I have removed it
I have manually verified that mounts are no longer being leaked (so pinkerton cleanup "device or resource busy" #38 is fixed)
networking setup is now handled by libcontainer and has been removed from containerinit
since containers are now started as direct children of flynn-host, we could stop passing FDs over unix sockets, but that would break restarting flynn-host or updating it (the containers will get re-parented to init, and we will lose the FD references).

/cc @titanous @josephglanville

titanous · 2016-07-03T01:37:54Z

host/flynn-init/main.go

+		if err := factory.StartInitialization(); err != nil {
+			log.Fatal(err)
+		}
+	}


Is there a reason to put this in init instead of main?

titanous · 2016-07-03T01:45:53Z

This is awesome! Have you tried removing the vendored libvirt client code?

titanous · 2016-07-03T01:46:12Z

Looks like we'll probably need some minor docs updates.

titanous · 2016-07-03T02:59:41Z

host/Tupfile

@@ -2,6 +2,5 @@ include_rules
 : |> sed 's/{{TUF-ROOT-KEYS}}/@(TUF_ROOT_KEYS)/g' cli/root_keys.go.tmpl > %o |> cli/root_keys.go
 : cli/root_keys.go |> !cgo |> bin/flynn-host
 : bin/flynn-host |> gzip -9 --keep bin/flynn-host |> bin/flynn-host.gz
-: |> !go ./flynn-init |> bin/flynn-init
+: |> !cgo ./flynn-init |> bin/flynn-init


This will break containerinit if the rootfs doesn't have libc. Perhaps we should put the libcontainer part in flynn-host instead?

lmars · 2016-07-03T20:52:49Z

@titanous comments addressed (in terms of docs, I just updated this line in the architecture docs).

titanous · 2016-07-03T21:51:20Z

host/libcontainer_backend.go

+// container, taken from:
+// https://github.com/opencontainers/runc/blob/v1.0.0-rc1/libcontainer/SPEC.md#security
+var defaultCapabilities = []string{
+	"CAP_NET_RAW",


This list does not appear to match the link.

In what way do they differ? I just took the ones with Enabled: 1 set.

Ah, I missed the second column. That makes sense.

lmars · 2016-07-06T13:44:13Z

Updates are not currently working because all containers are exiting when the old flynn-host process dies.

I initially thought flynn-host was killing them on shutdown but this also happens when it dies abruptly with SIGKILL, so it must be the kernel which kills the containers.

The likely suspect is the ParentDeathSignal parameter, but so far I have not been able to configure things so that the containers do not exit (I have tried modifying that config variable, removing all prctl calls, making a prctl call to reset it in containerinit, but none of those approaches prevent the containers from dying).

I am investigating further.

lmars · 2016-07-08T13:56:58Z

The containers were previously exiting when the parent flynn-host daemon exited because they were getting a TTY hangup signal due to the fact that flynn-host opened the console, which subsequently closed when it died.

I have changed things so that only TTY jobs get a console (f5d6b84) which means those jobs will not survive an update to flynn-host, but those jobs are usually interactive jobs (e.g. flynn run bash) so I don't think we need to worry about them.

I have also fixed job resurrection so that it only happens for jobs which aren't actually running (c7d703d), and although that could be changed on master I am bundling it in here as I hope we can merge this.

The last few test runs have had some failures but they are all tests which fail intermittently on master, and are mostly controller deployment related (something which I will be refactoring soon).

@titanous @josephglanville this is ready for a review.

titanous · 2016-07-08T21:57:17Z

host/libcontainer_backend.go

-					Source: lt.FSRef{Usage: "65535"}, // 64MiB
-					Target: lt.FSRef{Dir: "/dev/shm"},
-				},
+	ifaceName, err := netutils.GenerateIfaceName("veth", 4)


Should this be moved into the else block below?

Signed-off-by: Lewis Marshall <lewis@lmars.net>

The cgroup-lite package is responsible for mounting the cgroup controllers at /sys/fs/cgroup/{controller}, which flynn-host expects to be mounted. Signed-off-by: Lewis Marshall <lewis@lmars.net>

Signed-off-by: Lewis Marshall <lewis@lmars.net>

To make it easy to identify jobs in the output of `ps`. Signed-off-by: Lewis Marshall <lewis@lmars.net>

Flannel for example needs this to setup vxlan devices. Signed-off-by: Lewis Marshall <lewis@lmars.net>

Signed-off-by: Lewis Marshall <lewis@lmars.net>

lmars · 2016-07-09T18:43:27Z

@titanous so I went on a small TTY adventure today and eventually realised the issue was that not assigning a console to the container meant that when containerinit opened a PTY to hand to the job, it ended up having that PTY as its controlling terminal, so then exec'ing the job with Setctty for that same terminal would fail with EPERM because terminals can only be controlling for a single session.

Long story short, updating github.com/kr/pty to include this change fixed that, and means there is now no need to create consoles in flynn-host (218e56e), so therefore no TTY hang ups when flynn-host exits.

Ready for another review 😄.

titanous · 2016-07-09T20:58:43Z

LGTM!

josephglanville · 2016-07-09T23:03:16Z

LGTM. 🎉

lmars · 2016-07-10T23:24:51Z

woot!

temujin9 · 2016-07-11T01:35:41Z

Woot indeed. Can I get a head's up when this releases?

titanous · 2016-07-11T12:03:34Z

The latest nightly has it.

lmars force-pushed the host-libcontainer-runtime branch from e7630d9 to 555e7d5 Compare July 3, 2016 00:46

titanous reviewed Jul 3, 2016
View reviewed changes

lmars force-pushed the host-libcontainer-runtime branch from 555e7d5 to ef8b039 Compare July 3, 2016 20:50

titanous reviewed Jul 3, 2016
View reviewed changes

titanous mentioned this pull request Jul 6, 2016

test: VolumeSuite.TestVolumeTransmitAPI failing to delete volume #3045

Closed

lmars force-pushed the host-libcontainer-runtime branch 3 times, most recently from da3f1b1 to df57576 Compare July 7, 2016 01:27

lmars mentioned this pull request Jul 7, 2016

Support Ubuntu 16.04 #2789

Closed

titanous mentioned this pull request Jul 7, 2016

More disk space issues #3075

Closed

lmars force-pushed the host-libcontainer-runtime branch from df57576 to c7d703d Compare July 8, 2016 11:59

titanous reviewed Jul 8, 2016
View reviewed changes

lmars added 6 commits July 9, 2016 12:55

vendor: Add github.com/opencontainers/runc/libcontainer

580347a

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Replace libvirt-lxc backend with libcontainer

f2bcc50

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Assign a console to containerinit

b446546

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Use libcontainer to setup bind mounts

ded6a59

Signed-off-by: Lewis Marshall <lewis@lmars.net>

pinkerton: Fix rootfs mount leak

7554695

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Remove nsumount hack

cb7f776

Signed-off-by: Lewis Marshall <lewis@lmars.net>

lmars added 18 commits July 9, 2016 12:55

host: Avoid network interface name collisions

0f7c53d

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Add defaultMemory constant

49bc14e

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Move libcontainer init code into flynn-host

48a9e06

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Restrict the list of container capabilities

b3709ef

Signed-off-by: Lewis Marshall <lewis@lmars.net>

all: Remove libvirt references

722bb1d

Signed-off-by: Lewis Marshall <lewis@lmars.net>

all: Depend on cgroup-lite package

7dc0f80

The cgroup-lite package is responsible for mounting the cgroup controllers at /sys/fs/cgroup/{controller}, which flynn-host expects to be mounted. Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Move libcontainer init code into main()

7941c29

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Fix restoring container state

8ac4fea

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Pass job ID as argument to containerinit

4639211

To make it easy to identify jobs in the output of `ps`. Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Give host network jobs CAP_NET_ADMIN

555b9bf

Flannel for example needs this to setup vxlan devices. Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Only create a console for TTY jobs

bbeb787

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Set rlimits using libcontainer config

b119401

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Fix using --no-resurrect flag

4a393e0

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Only resurrect jobs if they are not running

f754144

Signed-off-by: Lewis Marshall <lewis@lmars.net>

vendor: Update github.com/kr/pty

f26b6e2

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Don't assign consoles to TTY jobs

218e56e

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Ensure cgroup subsystems are mounted

bea0afb

Signed-off-by: Lewis Marshall <lewis@lmars.net>

host: Move network interface name generation

b82b909

Signed-off-by: Lewis Marshall <lewis@lmars.net>

lmars force-pushed the host-libcontainer-runtime branch from c7d703d to b82b909 Compare July 9, 2016 18:29

titanous mentioned this pull request Jul 10, 2016

Container assigned invalid IP #2737

Closed

lmars merged commit 17d3592 into master Jul 10, 2016

lmars deleted the host-libcontainer-runtime branch July 10, 2016 23:24

This was referenced Jul 13, 2016

Near zero-downtime production upgrades #3104

Closed

Trying to update an already up-to-date cluster breaks it #2851

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

host: Replace libvirt-lxc backend with libcontainer #3030

host: Replace libvirt-lxc backend with libcontainer #3030

lmars commented Jul 1, 2016 •

edited by titanous

lmars commented Jul 3, 2016

titanous Jul 3, 2016

titanous commented Jul 3, 2016

titanous commented Jul 3, 2016

titanous Jul 3, 2016

lmars commented Jul 3, 2016

titanous Jul 3, 2016

lmars Jul 3, 2016

titanous Jul 3, 2016

lmars commented Jul 6, 2016

lmars commented Jul 8, 2016

titanous Jul 8, 2016

lmars commented Jul 9, 2016

titanous commented Jul 9, 2016

josephglanville commented Jul 9, 2016

lmars commented Jul 10, 2016

temujin9 commented Jul 11, 2016

titanous commented Jul 11, 2016

host: Replace libvirt-lxc backend with libcontainer #3030

host: Replace libvirt-lxc backend with libcontainer #3030

Conversation

lmars commented Jul 1, 2016 • edited by titanous

lmars commented Jul 3, 2016

titanous Jul 3, 2016

Choose a reason for hiding this comment

titanous commented Jul 3, 2016

titanous commented Jul 3, 2016

titanous Jul 3, 2016

Choose a reason for hiding this comment

lmars commented Jul 3, 2016

titanous Jul 3, 2016

Choose a reason for hiding this comment

lmars Jul 3, 2016

Choose a reason for hiding this comment

titanous Jul 3, 2016

Choose a reason for hiding this comment

lmars commented Jul 6, 2016

lmars commented Jul 8, 2016

titanous Jul 8, 2016

Choose a reason for hiding this comment

lmars commented Jul 9, 2016

titanous commented Jul 9, 2016

josephglanville commented Jul 9, 2016

lmars commented Jul 10, 2016

temujin9 commented Jul 11, 2016

titanous commented Jul 11, 2016

lmars commented Jul 1, 2016 •

edited by titanous