Handling of default users/groups

[bgilbert]

We need a way to provide default users/groups which:

1. Exist in the root filesystem when Ignition runs.
2. Can be modified by Ignition.
3. Allows us to create new users/groups on upgrade. (We probably can't remove any on upgrade, since there might be inodes that reference them.)

CL and AH use a mix of nss-altfiles, which fails point 2; systemd-sysusers, which fails point 1; and hardcoded entries in /etc/{passwd,group}, which fails point 3.

It appears that the consensus approach is to switch to systemd-sysusers and drop nss-altfiles:

• CL: Specifying group "systemd-journal" in cloud-init doesn't work bugs#312 (comment), Cannot add user to systemd-journal in ignition bugs#1920
• AH: Drop requirement on nss-altfiles, use systemd sysusers rpm-ostree#49, WIP: compose: Add experimental/sysusers option rpm-ostree#1679
• FCOS: manifest: Add static sysusers entries fedora-coreos-config#56

Complications:

1. systemd-sysusers isn't really intended to support non-system users, so for root and core we may need to do something else.
2. For system users, systemd-sysusers doesn't support some things, such as mismatched user/group IDs.
3. We'd need to force systemd-sysusers to run in the initramfs before ignition-files runs.
4. Deleting a default user/group from an Ignition config requires both deleting it (which Ignition can't do yet, Add a way to delete users/groups ignition#738) and masking the systemd-sysusers fragment so it isn't recreated. This will probably not be a common operation, except for the core user, for which see core user isn't added to groups when created via Ignition fedora-coreos-config#41.

[cgwalters]

3. We'd need to force systemd-sysusers to run in the initramfs before ignition-files runs.

Analogous to what we do for systemd-tmpfiles. In fact sysusers needs to run before tmpfiles:

$ grep After= /usr/lib/systemd/system/systemd-tmpfiles-setup.service
After=local-fs.target systemd-sysusers.service systemd-journald.service

[dustymabe]

Added a note to the f32 changes tracking ticket about f32 picking up sysusers.d: #372 (comment)

[lucab]

Self note: systemd-sysusers binary is already in the initramfs (without any service unit though).

[cgwalters]

xref coreos/rpm-ostree#2232

[travier]

There is a Fedora change to move user creation to sysusers.d format but this has not been updated in the docs yet (see docs PR).

Also, if packages use upstream %sysusers_create_package macro, this creates potential issues as this macro creates all users from all available config files. In Fedora this is currently the case for dnsmasq specfile:

⠤ Running pre scripts... console-login-helper-messages
dnsmasq.prein: Creating group fedora-coreos-pinger with gid 982.
dnsmasq.prein: Creating user fedora-coreos-pinger (Fedora CoreOS telemetry service user) with uid 982 and gid 982.
dnsmasq.prein: Creating group zincati with gid 981.
dnsmasq.prein: Creating user zincati (Zincati user for auto-updates) with uid 981 and gid 981.
dnsmasq.prein: Creating group dnsmasq with gid 980.

Apparently the systemd upstream macro got created on purpose that way, deprecating a previous macro which only consumed specific configuration files, not the whole set of entries, see systemd/systemd#8058 (comment).

EDIT(@lucab): added more refs.

[dustymabe]

we should bundle in #1201 as part of this.

[bgilbert]

coreos/ignition#1596 requests a workaround in Ignition until FCOS migrates to systemd-sysusers.

[travier]

Suggestion of plan in coreos/rpm-ostree#49 (comment)

[travier]

Dropping nss-altfiles

This is a proposal / list of steps to move away from using nss-altfiles in rpm-ostree based systems and rely on systemd-sysusers logic only. See coreos/rpm-ostree#49.

Context

Rpm-ostree based systems currently split the user and group definition in two parts, one coming from the image as part of /usr/lib/(passwd|group), which is read only, and one copied into /etc/(passwd|group) from /usr/etc/(passwd|group) (not sure exactly at what time exactly that happens) to make it writable for system administrators.

To "merge" those two sources, we use nss-altfiles (nss module configured in /etc/nsswitch.conf) to lookup users in both places:

$ cat /etc/nsswitch.conf | grep -E "^passwd:|^group:"
passwd:     files altfiles sss systemd
group:      files altfiles sss systemd

nss-altfiles adds complexity to the user/group lookup process and user/group creations on rpm-ostree systems and creates user experience issues:

• Adding a user to a group that is defined in the RO part (/usr/lib/group) requires that the group definition be copied first to the writable part (/etc/group) and then extended.
  • See: https://discussion.fedoraproject.org/t/how-can-i-add-myself-to-the-libvirt-group-in-fedora-silverblue/1412/2
• User/group lookups in the initramfs for the real system are harder to do and thus system user setups in Ignition phases have caveats.
  • core/dracut/ignition-ostree: add ignition-ostree-sysusers service  fedora-coreos-config#774

Where we want to go

The ideal future is that we are not using nss-altfiles anymore and instead all users and groups are defined by systemd sysusers configuration files and read from /etc/(passwd|group).

$ cat /etc/nsswitch.conf | grep -E "^passwd:|^group:"
passwd:     files sss systemd
group:      files sss systemd

New systems

New systems would start with a set of pre-created system users from the image like today (/usr/lib/passwd|group, copied to /etc) that would be extended as needed by:

• additional sysusers files installed by Ignition for system users
  • Ignition would have to pre-run the systemd sysusers config in the final root to be able to chown files to the right owners as needed
• additional interactive users created by Ignition

The /usr/lib/passwd|group files would then be completely unused on the system.

Updating existing systems

Existing systems must retain the current set of users and groups defined as well as their UIDs/GIDs.

In order to do that, we’ll need to create an intermediate step where we ship in the image the exact set of systemd sysusers configs matching the current set of defined users/groups.

Then we’ll be able to disable nss-altfiles in nsswitch.conf in an update and on the next boot with that update, the system should re-create the right users/groups in the /etc/(passwd|group) files, matching what was in /usr/lib/(passwd|group). This update must be a barrier release.

Once we are confident that this is working, we would be able to remove nss-altfiles completely from the image, and then start updating our user/group definition to newer Fedora standards as existing systems would retain their current set of users/groups while new systems would get the new ones.

[cgwalters]

The ideal future is that we are not using nss-altfiles anymore and instead all users and groups are defined by systemd sysusers configuration files and read from /etc/(passwd|group).

I wouldn't say that necessarily. I think in some cases we may actually want user/groups "locked" to the system, which could be done via JSON and nss-systemd.

It does seem clear to me that e.g. the libvirt group should switch to sysusers, so it ends up as mutable state.

But e.g. the zincati user? I don't think it needs to be local mutable state.

[HuijingHei]

Maybe we should also consider about ostree container ? Not sure if there is any difference about how we handle between OS and container.

[travier]

I should have said "all system user and groups are defined by systemd sysusers configuration files".

AFAIK, nss-systemd / systemd-userdb is more oriented towards interactive users than system users.

[travier]

We might be able to use pwck and grpck to do the migration on existing systems. Would need to be investigated.

[dustymabe]

We discussed this during the community meeting today:

12:53:25 dustymabe | #agreed The proposal to drop the use of
                   | nss-altfiles and replace with systemd-sysuers usage
                   | has no opposition.

[cgwalters]

Just a note that while systemd-sysusers makes sense for some things, I actually think we should use nss-systemd and hardcoded static JSON user records for some others.

[jlebon]

Replying to coreos/fedora-coreos-config#2670 (comment) here:

The use case where we need to run sysusers in the initramfs (#774) is when you want to declare a system user in a sysusers config file (written by Ignition to the real root) and also include a file in the Ignition config that is owned by that new system user. As the users will not be created in the real root when ignition-files runs, it would fail.

Hmm, but Ignition already has native support for users/groups manipulation via its .passwd section. I think doing this would require the files stage to do something like:

• create sysusers.d dropins from the .passwd section
• peek ahead at .storage.files and apply any sysusers.d dropins in there
• run systemd-sysusers
• process the rest of .storage.files

?

It's not clear to me that the "peek ahead" bit of complexity is worth it. Users are free to create sysusers.d dropins in their Ignition config, but for the case of ".storage.files owned by those new users", we could encourage the existing .passwd section instead and avoid having multiple ways to do things.

Also, I think we want systemd-sysusers to run before ignition-ostree-populate-var.service (which runs before ignition-files.service). So then the flow would look something like this:

• ignition-ostree-sysusers.service (runs systemd-sysusers)
• ignition-ostree-populate-var.service (runs systemd-tmpfiles)
• ignition-files.service (adds users/groups created in config)

The trickiest bit I think will be addressing the "removal" case. That's complication 4 in #155 (comment). Ignition does support shouldExist: false now to express "I don't want this user to exist", but there's still the "prevent it from being recreated in the real root" part. I'm not even sure if masking the sysusers.d dropin is correct since other users/groups might be created there. Ideally, the sysusers format would support an entry type for "don't create this user". I guess a temporary hack would be to copy the dropin and e.g. comment out that line? Yuck...

Maybe simplest for now is to not support this and try to get proper support for it upstream. Until then, users can always use .storage.files to mask a /usr dropin or override it with a dropin without the offending line.

[travier]

we could encourage the existing .passwd section instead and avoid having multiple ways to do things.

My preferred option would be that we change Ignition to generate and write sysusers files for system users to keep all system user records in that format.

Maybe that would not be directly a change in Ignition but in Butane sugar to avoid a spec change.

[jlebon]

My preferred option would be that we change Ignition to generate and write sysusers files for system users to keep all system user records in that format.

To be clear, you're suggesting that Ignition generates sysusers.d dropins from the .passwd section? That sounds fine; in that case, it could rerun systemd-sysusers pointing at its dropin before continuing to handling .storage.files. It shouldn't require a spec change.

[travier]

Only for system users (regular non system users would still be created "manually" by Ignition).

We would still need a split of the files stage which is not ideal indeed.

[HuijingHei]

I think we want systemd-sysusers to run before ignition-ostree-populate-var.service (which runs before ignition-files.service)

This means to make sure all configs under sysusers.d can be applied before ignition-files.service, in case of .storage.files owned by the systemd sysusers?

[travier]

Summary of a side discussion

We've clarified the following steps, which are not in order as they are somewhat inter-dependent:

• Add a service in initrd that runs sysusers on the real root before we run Ignition's populate-var and files stages. This is mostly: core/dracut/ignition-ostree: add ignition-ostree-sysusers service  fedora-coreos-config#774
• Add a treefile option in rpm-ostree to turn off nss-altfiles support and the passwd / group files migration to /usr/lib.
• Build a FCOS image without nss-altfiles to confirm that all users are correctly populated.

System users on new systems

We can either:

• start with a mostly empty passwd & group files like we have today in /etc (i.e. not take the users created during the compose) and have the sysusers stage re-create all the users on first boot
• or instead use the passwd & group files from the compose as default passwd & group files and only create "new" users.

The second option is likely the safest as it has less potential for failures.

How to update existing systems

Existing systems require a migration step to "merge" the content of the user/group definitions in /usr/lib and the content of /etc/. We can either add a systematic sysusers stage to the initrd that runs on all boots (may be removed after a barrier release) or let the real root sysusers service create the users. The second option has the downside that if users added units that depend on those system users and that run before this sysusers stage, then those units will fail on this boot.

Support defining users in Ignition configs via sysusers config files

This will be split from this issue as this requires work in Ignition.

Instead of splitting the Ignition files stage in two, Jonathan suggested that we add a .passwd.sysusers section to the Ignition spec that would take sysusers config files as input. They would then be written to the disk before the new sysusers stage so that the users are properly created before the files stage.

We can then add a warning / error in Butane if sysusers files are included in the files section to guide users to write them in this dedicated sections instead.

[HuijingHei]

Add a treefile option in rpm-ostree to turn off nss-altfiles support and the passwd / group files migration to /usr/lib.

I might misunderstood, in this case /usr/etc/{passwd,group} content will be the same as /etc/{passwd,group} during build process? Maybe /etc/{passwd,group} will include custom user like core?

[travier]

Removed the F40 tag as this is not landing in Fedora 40.

[dustymabe]

Should we add the F41-Changes tag?

[travier]

I'm adding it to #1599

[HuijingHei]

Sorry for the delay, will put it to my plate.

[HuijingHei]

TODO list:

• Add a service in initrd that runs sysusers on the real root before we run Ignition's populate-var and files stages. This is mostly: core/dracut/ignition-ostree: add ignition-ostree-sysusers service fedora-coreos-config#774 (Note: this is to make sure all configs under sysusers.d can be applied before ignition-files.service, in case of .storage.files owned by the systemd sysusers)

• Ignition: As Jonathan suggested that we add a .passwd.sysusers section to the Ignition spec that would take sysusers config files as input. They would then be written to the disk before ignition-ostree-sysusers service so that the users are properly created before the files stage

• rpm-ostree: Add a treefile option in rpm-ostree to turn off nss-altfiles support and the passwd / group files migration to /usr/lib.

• Build a FCOS image without nss-altfiles to confirm that all users are correctly populated.

[LorbusChris]

Related discussion on the Fedora ML: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IKWECWMBWN2IDKLHK3Q2TZKVSVFTXUNA/