Pebble doesn't report errors starting startup-enabled services

[barrettj12]

I have the following simple "service" at root/log.sh:

#!/bin/bash
while true
do
  echo "Hi I'm alive"
  sleep 1
done

and the following Pebble layer:

services:
  log:
    command: /root/log.sh
    override: merge
    startup: enabled

If I don't have permission to run /root/log.sh:

$ /root/log.sh
bash: /root/log.sh: Permission denied

then I would expect Pebble to report an error when I go pebble run, but it doesn't report anything:

$ pebble run -v
2022-12-02T02:47:24.670Z [pebble] Started daemon.
2022-12-02T02:47:24.692Z [pebble] POST /v1/services 20.668906ms 202
2022-12-02T02:47:24.693Z [pebble] Service "log" starting: /root/log.sh
2022-12-02T02:47:24.693Z [pebble] Started default services with change 9.

[benhoyt]

Yeah, I've noticed this too, and it's unexpected not to get any feedback (unless you specify PEBBLE_DEBUG=1). It's actually more general than just permission denied errors -- it's any error starting the startup-enabled services when the daemon starts.

Do you think it should log an error, or wait (up to 1 second per service, I think it is) and exit pebble run with an error message and non-zero exit code?

[barrettj12]

It doesn't report No such file or directory errors either.

I think it should log an error right away - because errors like permission denied and no such file or directory are "unrecoverable" - trying again is not gonna help, some manual intervention is needed.

I'm not sure what Pebble's usual behaviour is, but I think exiting pebble run is probably not the right move - you still want the other services to run if possible, I presume?

[hpidcock]

I'm not sure what Pebble's usual behaviour is, but I think exiting pebble run is probably not the right move - you still want the other services to run if possible, I presume?

if the service is critical to the plan then it should have

on-failure: shutdown

[benhoyt]

This was actually fixed by the snapd state changes we pulled across, now that the task runner logs task failures: e494ff2#diff-e8b7c8da8654dde95838cfa0cc4b497cdd32050ea3a1bf0640e6404303aef0d6R294

Additionally, we do slightly more logging with #257.

We can consider behaviour changes like having "on-failure: shutdown" default services not starting up make Pebble not start. But that needs a spec and further considering. This issue is just about reporting the errors, which is now in place, so closing.