Orbit hang on shutdown

[zwass]

Orbit version: 1.6.0

Operating system: macos 13.1

---

🧑‍💻  Expected behavior

Orbit shuts down and restarts after update, regardless of the state of the subprocesses.

💥  Actual behavior

Orbit hangs while restarting after update. Osquery and Fleet Desktop seem to have shut down. Nudge is still running and it looks like Orbit is waiting for Nudge to exit before completing exit (which would allow the whole system to restart and get osquery and Desktop back up).

More info

From the logs:

2023-02-21T21:23:18-08:00 INF calling flags update
2023-02-21T21:23:48-08:00 INF calling flags update
2023-02-21T21:24:16-08:00 INF update detected target=orbit
2023-02-21T21:24:16-08:00 INF update completed target=orbit
2023-02-21T21:24:16-08:00 INF exiting due to successful update
2023-02-21T23:47:33-08:00 INF token TTL expired, rotating token
2023-02-22T00:59:15-08:00 INF token TTL expired, rotating token
2023-02-22T02:59:53-08:00 INF token TTL expired, rotating token
2023-02-22T05:00:18-08:00 INF token TTL expired, rotating token
2023-02-22T07:01:14-08:00 INF token TTL expired, rotating token
2023-02-22T09:02:09-08:00 INF token TTL expired, rotating token
2023-02-22T10:02:32-08:00 INF token TTL expired, rotating token
2023-02-22T11:03:02-08:00 INF token TTL expired, rotating token
2023-02-22T12:03:13-08:00 INF token TTL expired, rotating token

It looks like Orbit tries to shut down, but then Nudge keeps running.

ps aux | grep orbit
root             30836   0.0  0.1 36563096  43976   ??  SN   Tue01pm   0:12.23 /opt/orbit/bin/nudge/macos/stable/Nudge.app/Contents/MacOS/Nudge -json-url file:///opt/orbit/nudge-config.json
root              3132   0.0  0.1 35170044  26208   ??  SNs  Tue11am   3:10.16 /opt/orbit/bin/orbit/orbit

[zwass]

Debug archive collected from the hung Orbit process: profiles-2023-02-23T00-17-11.tar.gz

[zwass]

When I clicked "I understand" and Nudge closed, it seems like Orbit was able to complete the shutdown process.

My thoughts:

1. Nudge being open should not prevent Orbit from restarting as usual (ideally Nudge would stay open but Orbit could still restart).
2. Independent of Nudge, Orbit should always have a timeout (say 10s?) to force-stop all of the subprocesses and complete the exit. Osquery does similar by first trying a SIGTERM and then a SIGKILL.

[zwass]

Ideally we could just change this from cmd.Run() to cmd.Start(), but the docs say that Wait() also needs to be called to clean up resources. This probably isn't too much trickier, but might require a goroutine and a channel in order to make it interruptible for the exiting.

I'm not sure what implication that has on how often the Nudge process runs (in relation to Roberto's comment just above), so that should be considered as well.