Complete rework of lxc-fedora template

[ganto]

This is another reply to #1356. After a few evenings of scripting and many iterations, I finally have something usable. It's a heavily refactored version of the Fedora container creation template. I removed compatibility to unsupported Fedora releases including sysvinit and yum stuff and tried to get rid of as much code as possible.

On the other hand I added some new commandline arguments found in other templates:

  --mirror   : To set custom HTTP(s) Fedora download mirror
  --packages : List of custom packages to install into a new container
  --debug    : Run with shell script with 'set -x'

There are also some new environment variables which can be used to customize the template behaviour. See --help output.

Supports Fedora >=24. By default Fedora 25 will be installed except on Fedora hosts, where the host release is taken. To simplify the code path (or at least not make it more complex) all non-Fedora hosts will now use the LiveOS-image based bootstrap environment even when they would natively support rpm, yum or even dnf (e.g. Gentoo, CentOS).

The resulting Fedora container mainly runs systemd services by default (journald, networkd, resolved, logind). Fixes #270.

For users who still want to use the old template (which is still functional) to setup old Fedora containers for testing stuff, I renamed the original lxc-fedora template to lxc-fedora-legacy

[lxc-jenkins]

This pull request didn't trigger Jenkins as its author isn't in the whitelist.

An organization member must perform one of the following:

• To have this branch tested by Jenkins, use the "ok to test" command.
• To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

[brauner]

You need to adapt the required Makefile.ams and friends to the name change from lxc-fedora tolxc-fedora-legacy otherwise the build will fail (conf. Travis):

make[1]: *** No rule to make target `lxc-fedora-legacy', needed by `all-am'.  Stop.

[ganto]

Ya, I saw too. I adjusted the templates/Makefile.am but it doesn't seem to be happy yet. Will have a look at it tomorrow.

[brauner]

jenkins: test this please

[ganto]

I tried to build a new Fedora image with build-image (from lxc-ci) in a yakkety container. As the new lxc-fedora template would always use the bootstrap image when running on non-native hosts, it resulted in an error not happening with the old template (which uses yum on Ubuntu):

root@yakkety:/home/ubuntu/lxc-ci# bin/build-image https://github.com/ganto/lxc lxc-fedora HEAD fedora 25 amd64 default /home/ubuntu/lxc-fedora
[...]
 ==> Executing: "mkdir -p /build/containers/LXC_NAME/rootfs" in /
 ==> Executing: "cp /etc/lxc/default.conf /build/containers/LXC_NAME/config" in /
 ==> Executing: "env root_password=root /usr/share/lxc/templates/lxc-fedora --path /build/containers/LXC_NAME --rootfs /build/containers/LXC_NAME/rootfs --name LXC_NAME -R 25 -a x86_64" in /
Checking cache download in /var/cache/lxc/fedora/25-x86_64/rootfs ...
Downloading x86_64 rootfs for Fedora 25 ...
Non-Fedora host detected. Checking for bootstrap environment ...
Setting up new Fedora 25 (x86_64) bootstrap environment.
Syncing LiveOS squashfs image from archives.fedoraproject.org::fedora-enchilada ...
[...]
    418,148,352 100%    4.38MB/s    0:01:30 (xfr#1, to-chk=0/1)
Mounting LiveOS squashfs file system.
mount: squashfs/: mount failed: Unknown error -1

Error: Mount of LiveOS squashfs image failed
[...]

As far as I understood, the squashfs-tools package should be installed in the build container as it is listed in LXC_RUN_DEPENDENCIES['ubuntu']. Might this be a capabilities or AppArmor issue?

[brauner]

Is this the correct syntax for mounting a squashfs image? Shouldn't it be

mount -t squashfs -o loop "${cache}/install.img" squashfs/

?

[brauner]

Is your squashfs mount syntax correct?

[ganto]

Hmn, it works without specifying the filesystem on my Gentoo box. This is something I just copied from the old template. But I guess it's not wrong to explicitly define it.

[ganto]

The updated version still doesn't work with the stacked LXC builders in my lab environment. Do you have any CI tests for templates?

[brauner]

@stgraber will know more about the image builders. :)

[stgraber]

Other than lxc-ubuntu and lxc-ubuntu-cloud which are covered by lxc-test-ubuntu, we don't have commit level or pull request level CI of LXC templates. I'm also considering disabling lxc-test-ubuntu from our normal CI runs as it's by very far the biggest time and bandwidth user. We'd cut our CI time by more than half should we turn that one off.

So I'm definitely not looking at adding more standard CI for LXC templates. That being said, we do produce daily images from a whole bunch of templates with various configuration of those templates. Those images are built inside LXC containers using code in github.com/lxc/lxc-ci and are run against the current git master of the templates, so when a template gets broken that's usually how we catch it within 24 hours.

[ganto]

@stgraber Ok, I see. Thanks for the explanation.

In the meantime I found the origin of the error: The problem is the loop mount.

First I tried to add the loop device nodes to the lxc.cgroup.devices.allow list as mentioned e.g. in Allow loop mounting files inside LXC containers?. Unfortunately this doesn't seem to be enough. There were no /dev/loopX device nodes in the container and therefore the loop mount obviously still failed. I checked other templates which still have static device files created with mknod, but this doesn't work in a systemd container as /dev will be mounted as tmpfs.

After manually creating a /dev/loop0 device file, I was at least able to mount the squashfs image manually. However, when running the lxc-create, it would still fail with the same error. Only after creating a second device file /dev/loop1 I was able to successfully run the lxc-fedora template in a LXC container.

How is the loop device setup supposed to work in a systemd container? Or would you allow add such a "hack" to the build-image script for creating the lxc-fedora image?

[ganto]

Any thoughts and comments on this PR? Is the loopback issue a blocker?

[stgraber]

I'll need to find some time to take a closer look at the script, but I don't think the loopback issue is a problem at all. You are correct that systemd based container are lacking the /dev/loopX devices and that those must be manually created.

We in fact have code doing exactly that in our current https://github.com/lxc/lxc-ci build-image script.

[ganto]

Ok, thanks for your reply.

I still saw that other templates have dedicated support for btrfs subvolumes when creating the rootfs cache. I'm not using btrfs, that's why I didn't implement that yet. If this is something desirable I could have a look at it for sure.

[stgraber]

Sorry for the delay. Don't bother about btrfs, that's fine for now.

However, one thing that should be sorted out for new templates is security. It looks like this template is downloading over plain http and rsync without doing any kind of gpg checks on the payload.

Any chance you can rework things to rely on https for early bootstrap so that folks can't take over the container with an http man in the middle attack?

[ganto]

Any chance you can rework things to rely on https for early bootstrap so that folks can't take over the container with an http man in the middle attack?

Thanks a lot for you reply. Yes, sure, that's something that I missed completely. I tried to mimic the behaviour of the old template which primarily uses rsync and then added optional HTTP support. I guess I should make HTTPS(!) default and only allow rsync (with an appropriate security note) if someone is downloading the image over a slow connection. Let me check the options...

[stgraber]

@ganto sounds good. And yes, we have a number of templates that weren't written with security in mind... We're working on fixing those now.

[ganto]

I guess switching the entire communication to HTTPS by default is what I can do for now. There are no checksums published of the squashfs bootstrap image.

I still could try to find a Fedora release GPG key on the host and try to validate the signature of the downloaded fedora-repos RPM (containing the GPG keys used for the rootfs installation). But I fear this might not be so reliable.

[stgraber]

Yeah, sounds good. https to download the bootstrap bits which include the package manager and keyring used for package downloads after that.