Periodic failures due to "No such file or directory"

[bcwaldon]

Playing around with deis, I noticed that sometimes when you deis scale up, some of the containers wouldn't launch. Digging through the logs, I found this:

Sep 15 18:19:17 deis-1 fleetd[579]: INFO manager.go:222: Writing systemd unit madras-stickpin_v3.cmd.10-announce.service (1235b)
Sep 15 18:19:17 deis-1 fleetd[579]: INFO reconcile.go:274: AgentReconciler completed task: type=LoadUnit job=madras-stickpin_v3.cmd.10-announce.service reason="unit scheduled here but not loaded"
Sep 15 18:19:17 deis-1 fleetd[579]: ERROR manager.go:113: Failed to start systemd unit madras-stickpin_v3.cmd.10-announce.service: Unit madras-stickpin_v3.cmd.10.service failed to load: No such file or director
y.
Sep 15 18:19:17 deis-1 fleetd[579]: INFO reconcile.go:274: AgentReconciler completed task: type=StartUnit job=madras-stickpin_v3.cmd.10-announce.service reason="unit currently loaded but desired state is launch
ed"

Manually calling systemctl start madras-stickpin_v3.cmd.10-announce.service worked fine, so clearly there's some race condition here between writing the file to disk and starting it in systemd.

[bcwaldon]

This may be related to constant daemon-reloading of systemd. We should test this bug out on v0.9.0 since we've stopped reloading so much.

[yaronr]

@bcwaldon ETA?

[guruvan]

Had several units load in new 557.2 machines and couldn't get them out of state "dead" - /run/fleet had the files in there, but systemctl restart unit failed with no such file or directory errors. At the suggestion of @robszumski I did a systemctl daemon-reload and then reloaded the units, and that got them right up and running. (haven't had a chance to dig through the logs...much skydns to wade to find the right error)