ConditionNeedsUpdate= (systemd unit) will never trigger

[travier]

Describe the bug

As we keep timestamp of files at 0 (UNIX Epoch) in our ostree commits (https://ostreedev.github.io/ostree/repo/#content-objects), this condition in systemd unit will never trigger.

A quick grep on my system show the following potentially impacted services:

$ grep -Rni ConditionNeedsUpdate= /usr/lib/systemd
/usr/lib/systemd/system/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:18:ConditionNeedsUpdate=|/var

# Symlinks to the services above:
/usr/lib/systemd/system/sysinit.target.wants/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/sysinit.target.wants/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:18:ConditionNeedsUpdate=|/var

More investigation is needed to see if that's an issue or just a quirk.

Reproduction steps

N/A

Expected behavior

ConditionNeedsUpdate= "works"

Actual behavior

ConditionNeedsUpdate= "doesn't work"

System details

N/A

Butane or Ignition config

N/A

Additional information

See https://bugzilla.redhat.com/show_bug.cgi?id=2230187

[travier]

Investigation for ldconfig.service:

[Unit]
Description=Rebuild Dynamic Linker Cache
Documentation=man:ldconfig(8)

ConditionNeedsUpdate=|/etc
ConditionFileNotEmpty=|!/etc/ld.so.cache

DefaultDependencies=no
After=local-fs.target
Before=sysinit.target systemd-update-done.service
Conflicts=shutdown.target initrd-switch-root.target
Before=shutdown.target initrd-switch-root.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/ldconfig -X

By default, if unchanged, /etc/ld.so.cache is updated alongside OS updates (from /usr/etc/ld.so.cache).
This becomes a problem if a config is added to /etc/ld.so.conf.d and the cache is rebuild. Then it won't be updated anymore alongside OS updates.

[travier]

Investigation for systemd-journal-catalog-update.service:

[Unit]
Description=Rebuild Journal Catalog
Documentation=man:systemd-journald.service(8) man:journald.conf(5)
DefaultDependencies=no
Conflicts=shutdown.target
After=local-fs.target systemd-tmpfiles-setup.service
Before=sysinit.target shutdown.target systemd-update-done.service
ConditionNeedsUpdate=/var

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=journalctl --update-catalog
TimeoutSec=90s

From journalctl():

       --update-catalog
           Update the message catalog index. This command needs to be executed each time new catalog files are installed, removed, or updated to rebuild
           the binary catalog index.

so this is not great. Should be safe to "just" remove this condition an run on all boots.

[travier]

Investigation for systemd-hwdb-update.service:

[Unit]
Description=Rebuild Hardware Database
Documentation=man:hwdb(7) man:systemd-hwdb(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-remount-fs.service
Before=sysinit.target shutdown.target systemd-update-done.service
ConditionNeedsUpdate=/etc
ConditionPathExists=|!/usr/lib/udev/hwdb.bin
ConditionPathExists=|/etc/udev/hwdb.bin
ConditionDirectoryNotEmpty=|/etc/udev/hwdb.d/

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=systemd-hwdb update
TimeoutSec=90s

This means that files added to /etc/udev/hwdb.d are not added to the hwdb and manually updated hwdb are not updated anymore. This should not impact most users but this is not great. Should also be safe to run on all boots, or just before a reboot.

[travier]

One 'option' to fix some of those would be to add more to "ostree finalize staged" like we did for SELinux.

[travier]

Hum, I've just found: https://github.com/ostreedev/ostree/blob/445194921365d5552a052e8b91f51812799ac764/src/libostree/ostree-sysroot-deploy.c#L3403

[travier]

On a freshly booted system:

[core@cosa-devsh ~]$ ls -alh /etc/.updated 
-rw-r--r--. 1 root root 190 Jan  1  1970 /etc/.updated

Not sure how related: https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html

The comment in the ostree code does not match the behavior described in the man page.

[prestist]

We talked about this in today's meeting. Removing meeting label.

[dustymabe]

@travier has asked the rpm-ostree/ostree folks to look at this.

[cgwalters]

The PR in ostreedev/ostree#1631 links to ostreedev/ostree#1628
It does offhand seem like something was missed here.

This is probably best tracked against ostree, but I can't transfer the issue there.

[travier]

I'll make a new issue as I can't transfer it either.

[travier]

Moved to ostreedev/ostree#3069.

Locking the discussion here to keep it in a single place.

[travier]

Closing as this is not an issue (already fixed). See ostreedev/ostree#3069 & coreos/fedora-coreos-config#2725