DLPX-93890 LTS 24.04: test_add_remove_route crashes with dracut-install core file

[prakashsurya]

Problem

The "kdump-tools" service depends on local filesystem's being mounted, so that it can write to "/tmp". Even though that service's definitation states it should run "After=local-fs.target", it doesn't appear to also "Requires=local-fs.target", so it can end up running before the local filesystems are ready.

Solution

The solution here, is to add an override file for the service, to add the necessary "Requires=local-fs.target" declaration.

Testing Done

• git-ab-pre-push is here

I applied this change manually, and verified the service's dependencies now depend on the local filesystems.

• Before:

$ sudo systemctl list-dependencies kdump-tools --all
kdump-tools.service
● └─system.slice
●   └─-.slice

• After:

$ sudo systemctl list-dependencies kdump-tools --all
kdump-tools.service
● ├─system.slice
● │ └─-.slice
● └─local-fs.target
●   ├─export-home.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─systemd-remount-fs.service
●   │ ├─system.slice
●   │ │ └─-.slice
●   │ └─local-fs-pre.target
●   ├─tmp.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─var-crash.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─var-delphix.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─var-log.mount
●   │ └─system.slice
●   │   └─-.slice
●   └─var-tmp.mount
●     └─system.slice
●       └─-.slice

[prakashsurya]

FWIW, I think this is a bug in the upstream service definition.. and likely due to the fact that the new service on develop has DefaultDependencies=no.

• on release:

# /lib/systemd/system/kdump-tools.service
[Unit]
Description=Kernel crash dump capture service

[Service]
Type=oneshot
StandardOutput=syslog+console
EnvironmentFile=/etc/default/kdump-tools
ExecStart=/etc/init.d/kdump-tools start
ExecStop=/etc/init.d/kdump-tools stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

$ sudo systemctl list-dependencies kdump-tools --all | head -n30
kdump-tools.service
● ├─system.slice
● │ └─-.slice
● └─sysinit.target
●   ├─dev-hugepages.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─dev-mqueue.mount
●   │ └─system.slice
●   │   └─-.slice
●   ├─keyboard-setup.service
●   │ ├─system.slice
●   │ │ └─-.slice
●   │ └─local-fs-pre.target
●   ├─kmod-static-nodes.service
●   │ └─system.slice
●   │   └─-.slice
●   ├─open-iscsi.service
●   │ ├─iscsi-name-init.service
●   │ │ ├─system.slice
●   │ │ │ └─-.slice
    │ │   └─...
●   │ ├─system.slice
●   │ │ └─-.slice
●   │ ├─network-online.target
●   │ │ └─systemd-networkd-wait-online.service
●   │ │   ├─system.slice
●   │ │   │ └─-.slice
●   │ │   └─systemd-networkd.service
●   │ │     ├─netplan-ovs-cleanup.service

• on develop:

$ sudo systemctl cat kdump-tools
# /usr/lib/systemd/system/kdump-tools.service
[Unit]
Description=Kernel crash dump capture service
DefaultDependencies=no
After=local-fs.target
After=networking.service
Before=basic.target

[Service]
Type=oneshot
StandardOutput=journal+console
EnvironmentFile=/etc/default/kdump-tools
ExecStartPre=-/usr/share/kdump-tools/kdump_mem_estimator
ExecStart=/etc/init.d/kdump-tools start
ExecStop=/etc/init.d/kdump-tools stop
RemainAfterExit=yes

[Install]
WantedBy=basic.target

$ sudo systemctl list-dependencies kdump-tools --all
kdump-tools.service
● └─system.slice
●   └─-.slice

[sebroy]

I don't understand. The systemd.unit man page says:

   Before=, After=
       These two settings expect a space-separated list of unit names. They may be specified more than once, in which case dependencies for all
       listed names are created.

       Those two settings configure ordering dependencies between units. If unit foo.service contains the setting Before=bar.service and both
       units are being started, bar.service's start-up is delayed until foo.service has finished starting up.  After= is the inverse of Before=,
       i.e. while Before= ensures that the configured unit is started before the listed unit begins starting up, After= ensures the opposite,
       that the listed unit is fully started up before the configured unit is started.

As such, it follows that kdump-tools won't begin starting up until after local-fs.target has started. Is the problem actually that our ZFS root isn't mounted as part of local-fs.target?

[prakashsurya]

As such, it follows that kdump-tools won't begin starting up until after local-fs.target has started.

Then, why isn't local-fs.target listed as a dependency of kdump-tools.service ? I don't think it's a problem with local-fs.target..

[sebroy]

Then, why isn't local-fs.target listed as a dependency of kdump-tools.service ? I don't think it's a problem with local-fs.target..

I think you answered my question in your latest comment relating to DefaultDependencies=no, which affects how After= behaves with respect to early-boot targets. If not for that, I think After= would be sufficient?

[prakashsurya]

which affects how After= behaves with respect to early-boot targets.

yea.. I haven't verified, but perhaps kdump-tools is being started before local-fs.. and After= only works if the services/dependencies are started "in the same systemd transaction group" (not 100% what that means, but some searching around seems to imply this).. if we Require= it as well, it'll ensure local-fs is started when kdump-tools is started..

kind of speculating, but I think that's matching the behavior I'm seeing..