Support Ignition?

[kevinschoon]

Hi --

Thanks for your work on Linuxkit!

Have considered supporting CoreOS ignition? It seems like a good choice given that it has support for most major cloud providers, is fairly mature, and overlaps with a lot of the less developed components of Linuxkit such as loading metadata, formatting disks, etc. It's integration with systemd and CoreOS itself could be an obvious incompatibility but it seems fairly modular. I have some basic integrations working locally now.

I would be interested in putting together a PR but I wanted to see if you all had any thoughts or know of anything that might make it totally incompatible.

Thanks!

[SvenDowideit]

Or conversely https://cloud-init.io/, which its probably even more common?

[justincormack]

We did take a look at both a while back, and decided that at least initially it made more sense to have some simpler tooling.

Cloud Init just seemed large (Python) and complex, plus hard codes a lot of assumptions about the CLI tools to make it work in various distros, which might need lots of patching. It also assumes that the filesystem is writeable in a lot of places, and that you install packages at boot time. I think writing a small parser for the relevant parts of the config (indeed Ignition has one in Go) would be simpler than trying to "port" it.

I think Ignition probably makes more sense. There are some systemd specific things, and it is designed to run once only (not every boot) but that may well not matter as we do not persist state anyway. In effect it is a Go rewrite of cloud init. It is not as modular as what we have, but things like the disk initialisation do have more features (although its just shelling out to sgdisk see https://github.com/coreos/ignition/blob/master/internal/sgdisk/sgdisk.go). The downside versus what we have now is that if you use it you have to specify everything in the init config, while with the current model you can hard code much of the behaviour and you need very little from the config, mostly we are just using it for ssh keys.

I know IBM were looking at cloud init type things, not sure what they went with in the end cc @davefreitag

[SvenDowideit]

I'll just leave this here: https://github.com/rancher/os/tree/master/config/cloudinit

its actually what CoreOS used before they made ignition, and now we're maintaining it for RancherOS - we'll probably extract it to be reuseable again.

[kevinschoon]

I ended up modifying Ignition to support additional OEMs (Linuxkit) and then started work on a package for Linuxkit.

We would need to make a few changes to support loading metadata in Linuxkit:

• Enable the kernel option CONFIG_FW_CFG_SYSFS for qemu support.
• Add the -fw_cfg name=opt/com.coreos/config,file=<IGNITION_CONFIG_FILE> to the qemu CLI wrapper.
• Update the metadata tool to support generating Ignition JSON.

If you guys think this is worth pursuing I can open up a PR and implement the necessary changes.

[justincormack]

• adding CONFIG_FW_CFG_SYSFS seems fine
• adding a way to set config with qemu seems fine, but we may want to use -data to set it as we do with hyperkit?

What would the third part do?

[kevinschoon]

Hi -- Sorry for the delay!

Re -fw_cfg:
I added a flag to Linuxkit that just allows support for passing arbitrary options to qemu which seems like the most flexible option. I am unsure about the Hyperkit flag you mention.

Re third part:
I realize this isn't strictly related as CoreOS has a another tool for collecting metadata and the existing one in Linuxkit should be good enough, so that I can be ignored.

After doing more research and updating my code (linked above) it seems we would need to implement something that roughly looks like (just as a temporarily shim shell script at /init):

/switch_root is the binary compiled from pkg/init/init.go
/oem is a hard coded file with an Ignition --eom option

/init:

#!/bin/sh
set -x

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# mount proc filesystem
mount -n -t proc proc /proc -o nodev,nosuid,noexec,relatime

# mount sysfs
mount -n -t sysfs -o nodev,noexec,nosuid sysfs /sys

# load devices
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

# We need link-local addresses
# for collecting userdata on
# most oem e.g. EC2
ip link set eth0 up

function recovery() {
  echo "Warning! Init failed, you have been dropped to a recovery shell!"
  exec /bin/sh
}

/usr/bin/ignition --stage disks --oem $(cat /oem)

[[ $? -eq 0 ]] || recovery

# Call init.go
exec /switch_root

# Second --stage files ignition commands happens after init

After the --stage disk we need to run --stage files before starting containers in /containers/onboot.

/etc/init.d/000-ignition:

#!/bin/sh

/usr/bin/ignition --oem $(cat /oem) --stage files

[rn]

is this still an issue, seem to have inactive for a long time, marking it as will be closed so it'll be closed in a week or so