How to run /etc/rc.d/rc.local as the last Linux bootup task?

[aki-k]

systemd version the issue has been seen with

239-68.el8_7.4

Used distribution

Rocky Linux 8.7

Linux kernel version used

6.2.10-100.fc36.x86_64 (host kernel, LXC container)

CPU architectures issue was seen on

x86_64

Component

other

Expected behaviour you didn't see

On Rocky Linux 8.7, the file /etc/rc.d/rc.local contains the following lines:

# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.

How to run /etc/rc.d/rc.local as the last Linux bootup task? On Rocky Linux 8.7 it fails to do the task that it
has always served. If it can't be made to run as the last task in Linux bootup, please remove rc-local.service
from systemd to avoid confusion.

Unexpected behaviour you saw

# cat /etc/rc.d/rc.local

Steps to reproduce the problem

# cat /etc/rc.d/rc.local

# rpm -qf /etc/rc.d/rc.local
systemd-239-68.el8_7.4.x86_64

Additional program output to the terminal or log subsystem illustrating the issue

No response

[poettering]

There's no such concept of "last" in systemd, because it makes no sense, it cannot be fulfilled in an event based system, or when more than one such service exists. Add explicit ordering deps instead.

Also rc.local is obsolete. Just write a proper service.

FInally, this is a bug tracker, not a support forum, as the form should have made abundantly clear...

[aki-k]

Why does systemd still carry it if it doesn't do what it used to do pre-systemd? (I mean rc-local.service)

[dtardon]

Why does systemd still carry it if it doesn't do what it used to do pre-systemd? (I mean rc-local.service)

Because rc.local scripts typically aren't that picky regarding their run position and work just fine. We even add After=network-online.target in RHEL to cover scripts that need network. And I'm pretty sure that the people who still insist on using it would shout at us very loudly if rc-local.service were removed...

[poettering]

Why does systemd still carry it if it doesn't do what it used to do pre-systemd? (I mean rc-local.service)

It does something quite similar.

But yeah, we should consider killing it now.

[aki-k]

@poettering Lennart, would you accept this kind of change to rc-local.service to keep it in systemd? This way rc.local is ran after crond in both multi-user.target and graphical.target, so at quite late stage before reaching the target:

[Unit]
After=crond.service

[Install]
WantedBy=default.target

# chmod 700 /etc/rc.d/rc.local
# systemctl enable rc-local.service

(I also removed /usr/lib/systemd/system-generators/systemd-rc-local-generator and created a symlink in its place to /dev/null.)

[poettering]

@poettering Lennart, would you accept this kind of change to rc-local.service to keep it in systemd.

No, I would not. See mailing list thread:

https://lists.freedesktop.org/archives/systemd-devel/2023-April/049038.html

[aki-k]

How about this config? I'm just trying to help keep rc.local alive in systemd. As Mr. Tardon wrote,
there are people that want to use it.

/etc/systemd/system/rc-local.service.d/override.conf:

[Unit]
After=systemd-user-sessions.service

[Install]
WantedBy=default.target

mkdir /root/systemd
mv /usr/lib/systemd/system-generators/systemd-rc-local-generator /root/systemd/
ln -s /dev/null /usr/lib/systemd/system-generators/systemd-rc-local-generator

chmod 700 /etc/rc.d/rc.local
systemctl enable rc-local.service

[poettering]

How about this config? I'm just trying to help keep rc.local alive in systemd. As Mr. Tardon wrote,
there are people that want to use it.

Why though? This should die, not kept alive "just because".

(Why do you mask the generator? That's entirely redundant. The generator will pull in the thing if the script exists, you really don't need to mask it, or move it away or anything like this. It's entirely redundant.)

[aki-k]

Why do you mask the generator? That's entirely redundant.

I don't know all the logic in systemd for it, but I noticed it puts rc-local.service into /run/systemd for multi-user.target.wants.

[poettering]

Why do you mask the generator? That's entirely redundant.

I don't know all the logic in systemd for it, but I noticed it puts rc-local.service into /run/systemd for multi-user.target.wants.

Yes it does. And thus enables it.

[aki-k]

But shouldn't it put it only for the default.target, be it multi-user.target or graphical.target ?

[poettering]

graphical.target pulls in multi-user.target, hence hooking it into multi-user.target is enough.

[aki-k]

Why though? This should die, not kept alive "just because".

I've already written the one reason why I want to use it. It's simple to run one-off commands and know that every other service is already running. You don't have to figure out the dependencies to other services.

Btw, why didn't systemctl enable rc-local.service warn me about the circular dependency problem when I used After=systemd-update-utmp-runlevel.service ? I only noticed it at the next reboot.

[dtardon]

I've already written the one reason why I want to use it. It's simple to run one-off commands and know that every other service is already running.

As has already been said, there's no such point in systemd world. Hence, the handling of rc.local in systemd is "best effort". It usually works; if it doesn't, it's your call to add the necessary dependencies locally.

Btw, why didn't systemctl enable rc-local.service warn me about the circular dependency problem when I used After=systemd-update-utmp-runlevel.service ? I only noticed it at the next reboot.

Because circular dependencies can only be discovered when a transaction is actually run.

[dtardon]

How about this config? I'm just trying to help keep rc.local alive in systemd. As Mr. Tardon wrote, there are people that want to use it.

It is alive and people can use it, with caveats.

[smod]

As has already been said, there's no such point in systemd world.

That's not entirely true, there is a org.freedesktop.systemd1.StartupFinished signal, but obviously rc-local service cannot blindly wait for it since it would block itself.

But there is a way to "do something as soon as startup is finished", using D-Bus.

[dtardon]

As has already been said, there's no such point in systemd world.

That's not entirely true, there is a org.freedesktop.systemd1.StartupFinished signal,

AFAICS that covers the startup of PID1, not the system, i.e., it's issued before the default target is even queued.

But there is a way to "do something as soon as startup is finished", using D-Bus.

There isn't, really. Even if this worked as you thought, rc-local.service is a part of the initial transaction too...

[dtardon]

But shouldn't it put it only for the default.target, be it multi-user.target or graphical.target ?

That's completely wrong thing to do. default.target is usually set to either multi-user.target or graphical.target on a normal boot, but it's not limited to these. Apart from being set manually, it can also be redirected by a generator. The cases of the latter I know of are single-user mode, offline update and SELinux relabeling. I'm pretty sure rc.local should be run in neither of those cases.

[aki-k]

default.target is usually set to either multi-user.target or graphical.target on a normal boot

I think this works when changing the default target:

systemctl disable rc-local.service
systemctl set-default multi-user.target # (or graphical.target)
systemctl enable rc-local.service

[arvidjaar]

AFAICS that covers the startup of PID1, not the system

StartupFinished should be issued after initial queue becomes empty, so for the practical purposes it is "after all units have been started". I was sure that OnStartupSec timer is relative to that event, but that seems to be relative to systemd start indeed.

[smod]

AFAICS that covers the startup of PID1, not the system, i.e., it's issued before the default target is even queued.

See manager_check_finished() in src/core/manager.c, looks compliant to what is documented: when the job queue becomes empty for the first time, "the boot is finished" (whatever semantic is put on this is another story).

There isn't, really. Even if this worked as you thought, rc-local.service is a part of the initial transaction too...

That's what I meant by "rc-local service cannot blindly wait for it since it would block itself.", but it can fork et detach something that does the job and immediately exit 0 to report readiness.

Just clarifying here for Google, there is a parallel discussion on the ML anyway.