rkt: lifecycle management

[philips]

We need to define how rkt knows how all of the processes running in a given stage1 have been destroyed and that the root filesystem can be cleaned up.

[jonboulle]

@vcaputo so braindumping here:

• rkt containers run in a standard location, $RKTDIR/containers/$UUID, where RKTDIR = the --dir flag to rkt (defaults to /var/lib/rkt) and UUID is generated at rkt run time (this is all the case in master)
• stage1 creates each app as a systemd service with Type=simple (i.e. not oneshot), potentially with Restart= policies depending on AppManifest
• stage1 creates (via the stage1 rootfs tarball and/or the stage1 app-> systemd conversion process) shutdown hooks so that when the app(s) finally exit (through failure or success), the shutdown target triggers a process that collects status information (effectively for i in $apps; systemctl status $app; done) in a known location in the stage1 rootfs (my thinking is having a dedicated rkt dir/file in the stage1, so for example /var/lib/rkt/containers/abc1234def/stage1/rkt/status)
• rkt status $UUID is implemented to first check if the given container is running (and if so, effectively do a systemd-nspawn systemctl status $apps), and if not, it parses this status output file

Then rkt gc is implemented separately to garbage collect old containers based on an age policy, etc.

[pierrebeaucamp]

Excuse me if I'm wrong, but I think I'm missing something here.

stage1 creates [...] shutdown hooks so that when the app(s) finally exit (through failure or success), the shutdown target triggers a process that collects status information

As far as I can detect it, reaper.service is only called onFailure, so the systemctl status loop is never launched on success. Therefor nothing is created in stage1/rkt/status

rkt status $UUID is implemented to first check if the given container is running [...]

rkt/status.go doesn't do anything so far

[vcaputo]

See exit-watcher.service for how stage1/reaper.sh gets invoked on success.

Have you observed nothing being created in stage1/rkt/status on successful exit? That would be a bug, and inconsistent with my observations.

We're still sorting out the status/gc/lifecycle side of rkt, the big question being: How does one distinguish active from inactive containers?

An exclusive advisory lock on the container's /var/lib/rkt/containers/$uuid directory bound to the rkt process' lifetime is the current direction we're exploring. The issue of the moment on this path is systemd-nspawn closes fds not in LISTEN_FDS so our directory lock fd is closed prematurely in stage1.

We've been using nspawn in stage1 out of convenience but it's increasingly becoming less so as things mature. In the course of fleshing out the lifecycle details nspawn may end up being replaced entirely by something specialized and minimal.

[jonboulle]

An exclusive advisory lock on the container's /var/lib/rkt/containers/$uuid directory bound to the rkt process' lifetime is the current direction we're exploring. The issue of the moment on this path is systemd-nspawn closes fds not in LISTEN_FDS so our directory lock fd is closed prematurely in stage1.

Mentioned briefly with code in #35 fwiw if anyone wants to dig into this further :-) (currently distracted by other priorities)

[pierrebeaucamp]

@vcaputo

See exit-watcher.service for how stage1/reaper.sh gets invoked on success.

Sorry didn't catched that one

Have you observed nothing being created in stage1/rkt/status on successful exit? That would be a bug, and inconsistent with my observations.

I had the understanding that stage1 would be terminated after stage2 was launched, therefor creating something in this directory. But it appears that no status is created when stage2 is starting (and the app gets executed) as well as when you kill the process.

We're still sorting out the status/gc/lifecycle side of rkt, the big question being: How does one distinguish active from inactive containers?

I see now why this is a problem, as I thought status would work in a different way. What about copying machinectl to stage1 as well so it can be used to get the status of the containers?

[vcaputo]

@pierrebeaucamp

I see now why this is a problem, as I thought status would work in a different way. What about copying machinectl to stage1 as well so it can be used to get the status of the containers?

Since rkt run is intended to function on non-systemd hosts as well we're not relying on systemd-specific facilities in the host for general functionality. We do have systemctl in stage1, but that's limited to interacting with the container's stage1 systemd instance. In the future we'll probably register with the host's systemd when available for improved integration, but that doesn't preclude the need for good general solutions.

I've put together a hacky PR which gives us both working advisory locks and a recorded container pid here: #244

This is not an attractive long-term solution but it does facilitate primitives for gc, list, status, etc. I think it's a reasonable intermediate step enabling movement on the other pieces.

[philips]

@vcaputo Is there a document on the current state of rkt gc/etc that we can point users at?

[vcaputo]

#414 is a first stab at an explanation

[jonboulle]

Fixed by #414, thanks @vcaputo