handle miscellaneous config tweaks

[wgwoods]

tl;dr: osbuild needs to be able to configure a variety of different services (dracut, sshd, chrony, cloud-init, waagent, network-scripts, udev, ...) that use simple text configuration files. Do we want do handle that kickstart-style (a stage for each service), ansible-style (generic file-manipulation stages), or take a hybrid approach (make a tool/module that does safe generic file tweaks and use that to implement per-service stages)?

Details: As outlined in #161, for parity with lorax-composer we need to be able to configure the following things:

• dracut (see Can't boot on some platforms, need way to modify initramfs contents #180)
• sshd (see Need to be able to configure ssh/sshd #181)
• WALinuxAgent (see vhd.ks)
• cloud-init (see vhd.ks, ami.ks, RCM's rhel-8*-ec2.ks)
• chrony (via kickstart timezone --nameservers - see google.ks)

These are five examples of miscellaneous high-level configuration tweaks where the actual implementation is basically just "set the value for this key in this {json,yaml,ini,key=val...}-format config file".

In kickstart-based image builds, anything that doesn't already have a kickstart command is handled in a %post scriptlet, using sed, cat << EOF, etc. Ansible does not ship modules for any of these tasks, but instead has generic modules for writing/modifying files: template, replace, template, lineinfile, blockinfile, ini_file, etc.

In an initial discussion last week, I preferred the generic approach, because years of dealing with user requests around kickstart commands and %post scripts make me think it's a bad idea to require osbuild developers to write a new stage for each new service or option our users need. It's too inflexible for users that have uncommon or proprietary services/tools, and those who are using osbuild but need something we don't provide will be stuck until we build and push a new osbuild release.

@larskarlitski and @teg were against the idea of generic stages, which is a sound principle, because the other big problem with Kickstart is that the "generic configuration" option (%post scripts) is far too generic and lets users make all kinds of wild and unintended changes that can completely break our tools or their systems in hard-to-debug ways. So we definitely need something that's safer than shell scripts.

It's also true that most of the things currently being handled in kickstart %post or Ansible playbooks don't need to be done during the image build. But some do, and we need some way to handle those things. So: thoughts?

[wgwoods]

(@ondrejbudai pointed out that we already have org.osbuild.chrony, so we don't need to worry about that one. Yay!)

[bcl]

In my use of ansible the lineinfile and blockinfile modules have been the most useful. They don't allow for uncontrolled scripting, but do allow users to tweak their configurations to that they actually can get things setup.

One other situation to consider is configuration of things that we can't anticipate, for example a user provided service with a config file that doesn't have useful defaults for whatever reason. So we wouldn't be able to write a stage for it, and they still need some way to manipulate the file.

[jillr]

I generally agree with @bcl. Ansible already gives you a solid template parser, no need to build something new. It looks like there's a fairly well known and concise list of config files to be concerned about covering so whitelisting what can be acted on in this stage and making sure those customization tasks can be really well tested should be fairly manageable and accessible to developers. Are there likely to be many user-provided services that need to be configured at build time? I'd expect most user config to happen post-build.

[goneri]

Something I like with the Ansible-scenario is that you can validate your configuration playbook/role against a running image. If you want to reset, you respawn an image. When you've done it right, you can just save the result. And integrate it in the image generation process.
The same thing with a regular script is no that easy.

[wgwoods]

I'd expect most user config to happen post-build.

Yeah, it seems pretty typical to have image configuration mostly happen at first boot. This makes sense, because there are a lot of common setup tasks can only be done after system is online in its target environment.

Are there likely to be many user-provided services that need to be configured at build time?

Well, if post-build user config happens after the system comes online, then build-time config has to handle anything that's required to get the system online, right?

This means handling anything that's required to set up a target system's storage devices and connect to the user's network, which can include things like:

• network storage devices (iSCSI, FCoE, ...)
• advanced storage config (mdraid, dmraid, mpath, ...)
• network authentication (802.1x)
• network policy requirements (firewall/service settings, etc)
• 3rd-party/proprietary network/storage drivers

...and so on. I wouldn't want to try to support each of them individually, at least.

[maxamillion]

@wgwoods are there any challenges that might prevent Ansible from being used for the build-time/pre-boot tasks? If so, I'm curious what they are and hopefully there are ways an Ansible plugin(s) could accommodate the requirements.

[bcl]

any challenges that might prevent Ansible from being used for the build-time/pre-boot tasks?

The general problem we've run into with using ansible for pre-boot configuration is that it isn't written with that in mind. It expects to be running on a booted system, not inside a chroot or directory tree without running services.

[pabelanger]

Coming from the CI works of openstack and now ansible. The pre-boot process (in the case of VMs and because SSHd was listed above) should basically be setup dns, enable SSH, add ssh keys. Then automation tooling like ansible can be then used to take over remote provisioning. Running ansible, and image boot time, for local connection should be fine today (as long as required dependencies are properly bootstrapped).

The most common deployments I've been involved with is create images generic enough to boot over multiple platforms, then any runtime configuration is done via glean or cloud-init. Then once only, do 2nd stage bootstrap.

[wgwoods]

@wgwoods are there any challenges that might prevent Ansible from being used for the build-time/pre-boot tasks? If so, I'm curious what they are and hopefully there are ways an Ansible plugin(s) could accommodate the requirements.

Hi @maxamillion - so, the short version is that osbuild assumes the target system is offline, and that you can't safely bring it online or run code from inside it¹. But Ansible (as Brian points out) assumes the target system is online, and makes changes by entering it and running code inside it.

So from my PoV the big challenges here would be:

1. determining what parts of existing Ansible modules are (or could be made) offline-safe, and which parts are online-only
2. figuring out a way to mark them or otherwise make Ansible/osbuild aware of the difference

For example: the lineinfile module seems like it could easily be made offline-safe - it doesn't really need to enter the target system and run code, it just modifies text files, right? Except: uid/gid lookups are not guaranteed to be reliable or correct on an offline system. So "group" and "owner" aren't offline-safe, although "uid" and "gid" would be fine.

So yeah. That's not a huge problem - it feels very fixable, I just don't know enough about Ansible internals or conventions to have a strong opinion on how to proceed. Offline-safe variant modules? An "offline operation" plugin? Would love to hear your thoughts on this.

---

¹There's a lot of good reasons for doing it this way - safety, reliability, reproducibility, cross-arch/cross-platform builds, etc - but the main thing is that we have to assume that we're building the image on a different system and in a different environment than the one where it'll be deployed. It's a lot like cross-compilation - we're using the host's tools to build an image for a different target.

[lavocatt]

@bcl do you know the status of this issue ?

[bcl]

@bcl do you know the status of this issue ?

I believe this is old and can be closed. AFAIK osbuild is taking a different approach to file editing now.