systemd: Use sysusers.d to create ws users

[martinpitt]

Move to using systemd's sysusers declarative files [1] for creating our
system users/groups. Arch already does that, Fedora moved to it since
Fedora 32 [2], and Debian supports it as well [3].

In debian/cockpit-ws.postinst, move the #DEBHELPER# block above the
statoverride, as the former now generates the user, and the latter needs
it.

Unfortunately Fedora/rpm's %attr does not really work with sysusers
files shipped upstream yet. The conf files are not installed yet during
%pre, but creating the users in %post is too late for the file
unpack phase, so cockpit-session would get the wrong permissions. Thus
duplicate the two sysusers config lines verbatim in %pre, which is at
least marginally better than calling useradd etc. programmatically.

Extend TestConnection.testWsPackage to remove the system users, reboot,
and validate that cockpit still works. This ensures correct sysusers.d
packaging across all distributions, as our normal CI images already have
the system users.

Fixes #15027

[1] https://www.freedesktop.org/software/systemd/man/sysusers.d.html
[2] https://fedoraproject.org/wiki/Changes/Adopting_sysusers.d_format
[3] https://manpages.debian.org/dh_installsysusers

---

• Rework this after PR Launch cockpit-session via socket activation on /run/cockpit/session #16808

We originally tried DynamicUser= in #16811, but that has been blocked for too long. IMHO this is a nice improvement already, and does not block moving to DynamicUser= in the future.

[martinpitt]

The arch failure is straightforward, already fixed/verified locally.

The two coreos failures are more serious: "error: While applying overrides for pkg cockpit-ws: Could not find group 'cockpit-wsinstance' in group file". It seems rpm-ostree install does not properly process the sysusers.d rpm script.

I moved it from %post to %pre, and that works. It's the right thing to do anyway, %post was an oversight. Yay tests!

This should also fix all the TF failures which use a fresh install instead of an upgrade. Nice to cover that case cleanly as well!

[martinpitt]

TF failures are still significant:

  Running scriptlet: cockpit-ws-282.1-1.20230103102400455339.pr18112.15   87/95 
Failed to open 'cockpit.conf', ignoring: No such file or directory
  Installing       : cockpit-ws-282.1-1.20230103102400455339.pr18112.15   87/95 
warning: group cockpit-wsinstance does not exist - using root

That's because %pre is actually too early -- of course the file does not exist yet. But later on it does get created:

Creating group 'cockpit-ws' with GID 990.
Creating user 'cockpit-ws' (User for cockpit web service) with UID 990 and GID 990.
Creating group 'cockpit-wsinstance' with GID 989.
Creating user 'cockpit-wsinstance' (User for cockpit-ws instances) with UID 989 and GID 989.

Investigating in a prepared fedora-37 VM:

rpm -e cockpit-ws cockpit; userdel cockpit-ws; userdel cockpit-wsinstance
rpm -i /var/tmp/build/cockpit-ws-*.rpm
# Failed to open 'cockpit.conf', ignoring: No such file or directory
# warning: group cockpit-wsinstance does not exist - using root
# Creating group 'cockpit-ws' with GID 982.
# Creating user 'cockpit-ws' (User for cockpit web service) with UID 982 and GID 982.
# Creating group 'cockpit-wsinstance' with GID 981.
# Creating user 'cockpit-wsinstance' (User for cockpit-ws instances) with UID 981 and GID 981.

But:

ls -l /usr/libexec/cockpit-session
# -rwsr-x---. 1 root root 58168 Jan  3 12:06 /usr/libexec/cockpit-session

This failure is an independent flake, I'll add a piggy-back commit here.

[allisonkarlitskaya]

See also #16811

[martinpitt]

@allisonkarlitskaya : Yes, I'm aware of this, I mentioned it in the description already. But I don't think this will make progress anytime soon.

[allisonkarlitskaya]

This looks good assuming everything passes. One comment for your consideration.

[allisonkarlitskaya]

I'd lean towards doing that in configure.ac. The only reason we need to do these ones here is to get the recursive path expansion (see the comment just above), but that's not an issue for these files.

[martinpitt]

I was actually vaguely aware of this, but I found it nice to keep all the systemd bits together in one file. I don't have a strong opinion about it, though.

[allisonkarlitskaya]

@allisonkarlitskaya : Yes, I'm aware of this, I mentioned it in the description already. But I don't think this will make progress anytime soon.

I totally agree. We're basically waiting on support in systemd which nobody seems to be interested in working on. I have no intention to block this PR in the meantime.

[martinpitt]

Hmm, the two IPA tests fail a little too often for my taste on R8.8 and C8S. A lot of it is noise due to the "3x affected" retries, but apparently not these. They reproduce well locally, I'll investigate them tomorrow.

Curiously these tests aren't even affected by the sysusers changes -- the packages are upgraded, and it's still exactly the same useradd users as before:

# getent passwd cockpit-wsinstance
cockpit-wsinstance:x:993:990:User for cockpit-ws instances:/nonexisting:/sbin/nologin

There should be a /etc/cockpit/ws-certs.d/10-ipa.cert, but isn't. And lo and behold:

audit: type=1400 audit(1672770758.901:4): avc:  denied  { create } for  pid=5969 comm="certmonger" name="10-ipa.key" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=0

With setenforce 0 both tests work fine, except for the unexpected SELinux message (with permissive=1).

Tests pass again when I drop the %sysusers_create_inline from the RPM, even though it ought to be a no-op if the users already exist.

Seems the RPM macro is broken:

/var/tmp/rpm-tmp.Ayc35I: line 27: warning: here-document at line 6 delimited by end-of-file (wanted `SYSTEMD_INLINE_EOF')

[allisonkarlitskaya]

could this be caused by a conflict with the existing users that we get from installing the package in the base image? in a certain sense we're not really properly testing the creation of new users here...

[martinpitt]

@allisonkarlitskaya : We test three cases now:

• Upgrades from the old useradd generated user, where the sysusers.d config should be a no-op (most tests in our CI). That's causing the subtle breakage of IPA certificate generation, which must be some weird side effect. I'm going to investigate it this morning.
• Fresh package installs with sysusers generated users from RPM %pre script (Testing Farm)
• sysusers generated users during boot after resetting /etc (TestConnection.testWsPackage)

[martinpitt]

LOL 🤣 The reason was that the macro is broken in RHEL 8 (see edited comment above), and thus the remainder of %pre was skipped. As a result, it was using the selinux-policy cockpit policy instead of our own. Fixed!

[allisonkarlitskaya]

I think the trouble we're seeing here is maybe a fundamental incompatibility between sysusers and having non-root-owned files installed by the package.

If the user gets deleted and later recreated, the uid on the filesystem might not be the same as the uid of the new user...

[martinpitt]

Ack, indeed. So let's shelve this and reconsider after #16808 -- if that ever lands, that would unblock the sysusers approach. Thanks for catching!

[martinpitt]

This is still very far out, and I don't want to stare at this on my /pulls page for that long. I sent the Debian packaging fix to #18339, and will close this one for now.