Systemd ignores After and Requires for my service

[ygoe]

systemd version the issue has been seen with

241 (I have no influence on that, it's part of the OS distribution, latest version)

Used distribution

Raspbian OS, Linux kernel 5.4.51

Expected behaviour you didn't see

My service is started after the bluetooth service has started.

Unexpected behaviour you saw

My service is started and fails. Later, the bluetooth service is first started, successfully. Later, my service is retried and now also succeeds.

Steps to reproduce the problem
Here's the script of my service:

[Unit]
After=bluetooth.service
Requires=bluetooth.service

[Service]
Type=simple
ExecStartPre=/usr/bin/sdptool add SP
ExecStartPre=/bin/sleep 0.5
ExecStartPre=/bin/hciconfig hci0 piscan
ExecStart=/usr/local/bin/my-service.py
ExecStartPost=/usr/bin/bluetoothctl discoverable on
Restart=always
RestartSec=15

[Install]
WantedBy=multi-user.target

[poettering]

This is a bug tracker for current upstream systemd versions, i.e. 245 is current, 241 is too old.

Also, the bug report is not useful, you provide no logs, no status output, no nothing, we have no idea what "bluetooth.service" does. Maybe it also uses Type=simple (i.e. is configured for no start-up notification), it's a black box for us.

[ygoe]

I can provide the full syslog of this boot. Do you need the early entries, too, or just the relevant ones related to these two services?

bluetooth.service is a default service on a Raspberry Pi. It's Type=dbus. I don't know what exactly it does, and it should not matter. My service needs to start after whatever bluetooth.service did. This is not the case although being specified so. Systemd will not even try to start bluetooth before starting my service. This is not what the documentation (and every other web resource) describes.

I have no reason to believe that there's a significant difference here between version 241 and 245, or even older ones as this config has been described for years on several websites. If Raspbian used the current Systemd version (which is impossible for me to try – and they probably won't accept Systemd issues and redirect me to here), I bet it would be the same.

[mbiebl]

$ systemctl cat bluetooth.service 
# /lib/systemd/system/bluetooth.service
[Unit]
Description=Bluetooth service
Documentation=man:bluetoothd(8)
ConditionPathIsDirectory=/sys/class/bluetooth

[Service]
Type=dbus
BusName=org.bluez
ExecStart=/usr/lib/bluetooth/bluetoothd
NotifyAccess=main
#WatchdogSec=10
#Restart=on-failure
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
LimitNPROC=1
ProtectHome=true
ProtectSystem=full

[Install]
WantedBy=bluetooth.target
Alias=dbus-org.bluez.service

This target is hard ware activated via

/lib/udev/rules.d/99-systemd.rules:SUBSYSTEM=="bluetooth", TAG+="systemd", ENV{SYSTEMD_WANTS}+="bluetooth.target", ENV{SYSTEMD_USER_WANTS}+="bluetooth.target"

I suppose you should hook up your service in bluetooth.target and not multi-user.target

[poettering]

I suppose you should hook up your service in bluetooth.target and not multi-user.target

no need given that he uses Requires= and After= on the service.

Maybe the ConditionPathIsDirectory=/sys/class/bluetooth is the issue though: not sure if the dir even exists before a bt device is plugged in, hence in that case bluetooth.service is simply skipped...

If so, this racy behaviour is explicitly configured in bluetooth.service and needs to be fixed there...

[mbiebl]

Isn't this problematic though, if the hardware isn't actually attached / ready. Trying to trigger the start of bluetooth.service might just result in bluetoothd being started too early.

[ygoe]

I tried changing the multi-user.target to bluetooth.target and that indeed helped. My service is now correctly started after Bluetooth is up. (Just for clarification: I disabled the service and re-enabled it to remove the old links to multi-user.target.)

Is there any explanation for this?

The Bluetooth device is on-board, it's not plugged in. It's always there.

[poettering]

Isn't this problematic though, if the hardware isn't actually attached / ready. Trying to trigger the start of bluetooth.service might just result in bluetoothd being started too early.

nah, it cannot be started too early. it's ordered after basic.target, (since it doesn't use DefaultDependencies=no) and thus will always be started only after dbus and stuff is up

[mbiebl]

If bluetooth is compiled as a module, it could be started before the bluetooth stack is ready, no?

[poettering]

Is there any explanation for this?

The Bluetooth device is on-board, it's not plugged in. It's always there.

my guess is that the bt stack is a kmod on your OS. Which means only as soon as that modules has been determined to be something to start (which might come after usb probing and such), and has been loaded the /sys file exists and because of the Condition check in the unit file it will then be skipped if started before.

quite frankly the condition should just not be there if the bt stack is not linked into the kernel itself. something to bring up with the bluez maintainers. the condition check is simply racy if the bt stack is added in only when used which might possibly be quite late

[poettering]

do things work if you leave things in multi-user.targer but drop the condition check from bluetooth.service?

[poettering]

another fix might be to add Wants=modprobe@bluetooth.service + After=modprobe@bluetooth.service so that the bt stack itself is loaded if needed, but if you do that the condition check is pretty pointless...

[mbiebl]

I don't get it though: we explicitly document bluetooth.target in systemd.special. Why do you think multi-user.target is a better target to hook his service into?

[poettering]

I don't get it though: we explicitly document bluetooth.target in systemd.special. Why do you think multi-user.target is a better target to hook his service into?

i don#t think that. what makes you think i do?

[mbiebl]

I suppose you should hook up your service in bluetooth.target and not multi-user.target

no need given that he uses Requires= and After= on the service.

Am I misreading you here?

[poettering]

I suppose you should hook up your service in bluetooth.target and not multi-user.target

no need given that he uses Requires= and After= on the service.

Am I misreading you here?

Well, but that was about @ygoe's service, not bluetooth.service.

I am pretty sure it's totally OK to have a service that runs as part of m-u.t, and still make use of bluetooth. if your service only does bt for some peripheral feature and primarily does some other stuff it might make sense to stick it in m-u.t and not in b.t, since the bt feature doesn#t matter too much.

All I am saying: even if bluetooth.service is usually activated via bluetooth.target doesn't mean it should be made impossible to also activate as dep of something typically pulled in via multi-user.target

[mbiebl]

And your other replies where you suggest other workarounds and keeping the service in multi-user.target instead of moving it to bluetooth.target.
Moving it to bluetooth.target seemed like the obvious correct solution to me. Anyway.

[poettering]

And your other replies where you suggest other workarounds and keeping the service in multi-user.target instead of moving it to bluetooth.target.
Moving it to bluetooth.target seemed like the obvious correct solution to me. Anyway.

I think the Condition check in bluetooth.service is just wrong. i'd consider that a bug in bluez, that should be fixed upstream.

I think what @ygoe is doing should just work, but the current upstream bluetooth.service makes that hard. I didn't really weigh into which target @ygoe should actually be using, there things aren't clear cut, and neither of the two target would be a "bug" to use, i.e. both should be fine to use, depending on the precise service you are defining.

Anyway, to summarize:

1. I think bluetooth.service should drop the condition, i.e. someone should file a bug upstream asking for that, maybe referring back to this issue.
2. In the meantime multiple work-arounds exist
   a. @ygoe's service can be moved to bluetooth.target, too
   b. The condition can be dropped locally via a drop-in
   c. The dep pair on modprobe@bluetooth.service could be added to bluetooth.target, locally (or upstream).

[poettering]

Either way, I think we can close this here, this is really between bluez and @ygoe's service I guess. In upstream there's nothing to change afaics.

[poettering]

/cc @holtmann

[ygoe]

Just for explanation, my service totally relies on Bluetooth. It opens an RFCOMM socket and waits for connections. If that fails, my service cannot work. And that's what happened: it failed and had to be restarted as per the service configuration.

I don't know much about targets. All of my previous services targeted multi-user, that seemed to be the go-to default for normal applications. Doesn't seem to be the case here. Targeting bluetooth has resolved my issue and I have no problem with that solution.

[poettering]

Anyway, I still think the condition really should go, it's simply racy. But that's a bluez issue. Closing here.