cli: Agent as init in rootfs images

[devimc]

Let kernel start the default init daemon (/sbin/init) and rid of
all related to systemd.
Currently we are tied to systemd because we have hardcoded it, we use
systemd even when we don't know if the image has it. Now that we can use
kata-agent as init process in rootfs images (kata-containers/agent#376), we
should use it, since we are responsible for any security flaw found in
kata-containers, it doesn't matter if we wrote that code or not (systemd).
The usage of kata-agent as init process brings performance improvements in
boot time and memory footprint.

Depends-on: github.com/kata-containers/tests#948

fixes #767

Signed-off-by: Julio Montes julio.montes@intel.com

[grahamwhaley]

Interesting change. IIU, that is conceptually quite a change - effectively turning off systemd in our .img images (by skipping it as init)?
We should discuss I think. Traditionally we have left systemd in the guest rootfs as init on the believe that customers may wish to add/change/ammend it. With this change, as your comment nicely says, we have not disabled that completely (they can add it back), but we have made it a little harder and less obvious.

So, questions:

• do you then intend to drop systemd from the rootfs .img file generation as well in osbuilder?
• Do you have boot-time numbers to compare - you can use the reportgen and grabdata.sh -t to get just boot time numbers for a before/after comparison for instance.
• does disabling systemd drop any features we use right now, or make anything else harder - maybe like enabling a debug console inside the rootfs? (if so, we'll need to fix those docs as well).

I'm all for optimisation and boot/speed improvements - just want to check we've looked at all the knock-on consequences.

/cc @egernst @mcastelino @WeiZhang555 @bergwolf for any thoughts.

[grahamwhaley]

And the CIs are really not happy. I'm seeing all sorts of odd fails, like the runtime is maybe just bust? Can you check those out to see if it was spurious or a real issue pls?
from 16.04, one of many:

Running command '/usr/local/bin/kata-runtime [kata-runtime --h]'
[kata-runtime --h]
Timeout: 60 seconds
Exit Code: 0
Stdout: NAME:
   kata-runtime - kata-runtime runtime

[jodh-intel]

Hi @devimc - I agree with @grahamwhaley, it would be great to get a bit more detail on this.

fwics, the problem with this change is that it assumes either the runtime's configuration.toml has been configured with the systemd options you show in the comment on this PR. Presumably that isn't the case in the CI test environment.

I think what you need to do is create an osbuilder PR that will effectively run the following when building an image where AGENT_INIT=no:

$ sudo chroot $ROOTFS systemctl set-default kata-containers.target
$ sudo chroot $ROOTFS systemctl mask systemd-networkd.service 
$ sudo chroot $ROOTFS systemctl mask systemd-networkd.socket

That will then create an image that does not require any systemd options to be specified in configuration.toml.

[WeiZhang555]

I think @grahamwhaley made a good question list 😄

I'm not very clear about the usage, if user want to use agent as init, they can also choose a custom image file, so what's the difference with that?

[jodh-intel]

Ping @devimc :)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@2931f8d). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #768   +/-   ##
=========================================
  Coverage          ?   65.29%           
=========================================
  Files             ?       87           
  Lines             ?    10604           
  Branches          ?        0           
=========================================
  Hits              ?     6924           
  Misses            ?     2983           
  Partials          ?      697

[amshinde]

@devimc Any updates?

[devimc]

@jodh-intel @amshinde sorry, nop, I haven't had time to run any metrics script, I'm working on the namespaces stuff

[raravena80]

@devimc ping :)

[raravena80]

@devimc nudge, any progress? 😄

[devimc]

/test

[devimc]

/test

[devimc]

/test

[devimc]

/test

[jodh-intel]

I think we need to be a little careful about the wording here as we will be using systemd by default.

Also, since we're removing the systemd masks here, shouldn't they be added to the osbuilder PR (or maybe we no longer need them at all)?

[devimc]

@jodh-intel

Also, since we're removing the systemd masks here, shouldn't they be added to the osbuilder PR (or maybe we no longer need them at all)?

we don't need them, these mask files are present if systemd is installed (AGENT_INIT=no in osbuilder)

[devimc]

@grahamwhaley do you know why metrics CI is not happy, was a regression detected?

[grahamwhaley]

Heh heh, let's call it a 'change', and not a regression... it looks like:

01:29:36 Report Summary:
01:29:36 +-----+----------------------+-------+-------+--------+-------+-------+-------+------+------+-----+
01:29:36 | P/F |         NAME         |  FLR  | MEAN  |  CEIL  |  GAP  |  MIN  |  MAX  | RNG  | COV  | ITS |
01:29:36 +-----+----------------------+-------+-------+--------+-------+-------+-------+------+------+-----+
01:29:36 | *F* | boot-times           | 95.0% | 73.5% | 105.0% | 10.0% | 71.9% | 77.8% | 8.3% | 1.8% |  20 |
01:29:36 | *F* | memory-footprint     | 85.0% | 71.2% | 115.0% | 30.0% | 71.2% | 71.2% | 0.0% | 0.0% |   1 |
01:29:36 | *F* | memory-footprint-ksm | 95.0% | 73.5% | 105.0% | 10.0% | 73.5% | 73.5% | 0.0% | 0.0% |   1 |
01:29:36 +-----+----------------------+-------+-------+--------+-------+-------+-------+------+------+-----+
01:29:36 Fails: 3, Passes 0
01:29:36 Failed

all items (boot time and memory footprint) shrank.
I'd recommend you try it locally using the report generator to confirm.

In conclusion - all the changes were in the 'good' direction - which is less of a worry than if they all went the other direction ;-)

[devimc]

@grahamwhaley thanks, I'll use the report generator

[devimc]

After running metric tests, I see that we'll have better numbers removing systemd

boot time

memory footprint

cc @kata-containers/runtime

[sboeuf]

@devimc please answer the questions you have been asked since it would help everybody to understand what is the purpose of this change. As mentioned by @WeiZhang555:

I'm not very clear about the usage, if user want to use agent as init, they can also choose a custom image file, so what's the difference with that?

[devimc]

@WeiZhang555

Sorry, I forgot that question

I'm not very clear about the usage, if user want to use agent as init, they can also choose a custom image file, so what's the difference with that?

There are no difference, but currently we are tied to systemd because we have hardcoded it, we use systemd even when we don't know if the image has it

func needSystemd(config vc.HypervisorConfig) bool {
	return config.ImagePath != ""
}

from my point of view and now that we can use kata-agent as init process in rootfs images (kata-containers/agent#376), we should use it, since we are responsible for any security flaw found in kata-containers, it doesn't matter if we wrote that code or not (systemd).
The usage of kata-agent as init process brings performance improvements (as was demonstrated). I consider we should use it by default, and let users choose their own poison by changing to another init process.

cc @sboeuf

[sboeuf]

Thanks @devimc, this makes sense to me. Several comments based on this:

• please make sure to highlight this properly in your commit message, particularly the fact that you would like to see kata-agent as the init process by default.
• systemd was pretty convenient for all the logging to the journal, are we going to loose this? And what about the tracing that @jodh-intel is enabling, will this continue to work?
• please bring that topic to the AC meeting and/or make sure all the key Kata stakeholders are agreeing on this. The PR is clearly simple, but this implies some big changes, and everybody needs to be on the same page.

[devimc]

@sboeuf

please make sure to highlight this properly in your commit message, particularly the fact that you would like to see kata-agent as the init process by default.

sure thing

systemd was pretty convenient for all the logging to the journal, are we going to loose this?

journalctl -b -t kata-proxy | grep "name=kata-agent" works, or what do you mean?

And what about the tracing that @jodh-intel is enabling, will this continue to work?

with kata-agent as init? I don't know, @jodh-intel any thoughts?

please bring that topic to the AC meeting and/or make sure all the key Kata stakeholders are agreeing on this. The PR is clearly simple, but this implies some big changes, and everybody needs to be on the same page.

I will do it

[devimc]

/test

[sboeuf]

@devimc

journalctl -b -t kata-proxy | grep "name=kata-agent" works, or what do you mean?

Well I think I might be confused, I was thinking we were pushing logs to the journal (inside the guest) from the agent, and those might not be available if systemd was not initializing the journal.

[bergwolf]

@devimc If users want speed and density, why not just use initrd with agent as the init process instead? I'm asking because I thought rootfs+systemd aims to provide more features (such as logging and other VM-like services). In your metrics tests, could you please add comparison with intird+agent-as-init?

[grahamwhaley]

Thanks for the graphs @devimc
The memory footprint one looks sane, with a reduction for the kata-agent.
The boot time one however we should probably try to reproduce. On the upside, kata-agent is definitely no worse - but, those 3 massive spikes on the systemd run sway the results - without those spikes the numbers would be much closer (but, kata-agent would still be a win it seems).
I've not seen spikes that bad before - what sort of machine did you run on/under? My suspicions are something like a laptop with a full UI running or inside a VM :-)

[jodh-intel]

Whatever we decide, since we offer these different image types, we need something like the following to help users decide what is most appropriate option for them as it's starting to get confusing (bad! :):

Image type	init	Boot speed	Image Size	Supports Factory?	Supports agent tracing?	Notes
rootfs	systemd	good	small	no	yes	Flexible as easy to customise
rootfs	agent	faster	small	no	yes	?
initrd	agent	fastest	smallest	yes	no	Not flexible
initrd	systemd	n/a	n/a	n/a	n/a	Not supported

[grahamwhaley]

Table of options and pros and cons, absolutely - we have been missing that for some time (probably since we added the factory as the 3rd option at least). I am of course going to suggest it gets added to the WIP User Guide

[jodh-intel]

@grahamwhaley - good call and done: https://github.com/kata-containers/documentation/wiki/UserGuide#images.

@devimc (and in fact anyone else! ;) - feel free to add to that document! ;)

[devimc]

@grahamwhaley

I've not seen spikes that bad before - what sort of machine did you run on/under? My suspicions are something like a laptop with a full UI running or inside a VM :-)

yes, a laptop with full UI

[devimc]

@bergwolf

Boot time

Memory footprint

[jon]

Quick summary of comments from discussion in the Arch Council call just now:

1. Most folks likes the concept -- thinner, lighter, less surface area is better -- many workloads don't really use systemd other than "because it's there"
2. Some workloads do require systemd -- they use it for the features it actually supports, Kata shouldn't try to clone those, so systemd must remain an option
3. In general, the gap for agent is that there's no way to kick off "init" services for things like a debug console or OpenTracing support

There seemed to no dissent (I won't call it consensus since there wasn't affirmative assent either :) for the idea that if 3 could somehow be addressed, this is a good idea.

One suggestion for how to tackle 3 is to leverage Agent's existing support for running and managing containers to... run and manage "init" services as static containers specified in the image. A strawman for how such a thing could be implemented:

1. Populate a well-known directory in the initrd, something like /etc/kata/init-rpcs with text-format protobufs (quick realization here -- outside Google text protobufs are probably a lot less common -- JSON proto serialization would also work) with "boot" RPCs
2. At startup, kata-agent processes each RPC as if it were received over AF_VSOCK
3. Any RPC that fails results in a panic, allowing fail-fast debugging of flaky bootstrapping.

This is borrowing heavily from the Kubernetes concept of Static Pods, although it goes one level higher in generality and lets the init behavior be a static set of RPCs. This wouldn't work if you wanted to do anything dynamic, although you could always Inception things and kick off a privileged bootstrap container that connects to the agent and dynamically issues RPCs. An alternative, if that level of generality isn't desirable, would be to populate the path with just OCI specs, at which point it really looks like Static Pods (or systemd services, for that matter).

[jodh-intel]

Great write-up @jon! Reading it has just reminded me that we in fact already have a very similar capability in the agent today - guest hooks:

• agent: add support for guest-hooks agent#365

[devimc]

Hi @jon thanks for the summary 😄