deis-logger stopped working

[nathansamson]

Earlier today I noticed that I lost app & platform logs for the last few days (since may the 15th).

Restarting deis-logger & deis-logspout has resolved the issue for now, but obviously it probably will return after a while...

core@deis-1 /var/lib/deis/store/logs $ journalctl --unit=deis-logger.service
-- Logs begin at Fri 2015-05-15 11:33:38 UTC, end at Mon 2015-05-18 18:33:46 UTC. --
May 16 02:47:30 deis-1 sh[22364]: 2015/05/16 02:47:30 Could not find app name in message:
May 18 17:52:21 deis-1 sh[22364]: 2015/05/18 17:52:21 Could not find app name in message:
May 18 18:18:31 deis-1 systemd[1]: Stopping deis-logger...
May 18 18:18:31 deis-1 sh[22364]: Shutting down...
May 18 18:18:32 deis-1 sh[22364]: time="2015-05-18T18:18:32Z" level="fatal" msg="Error response from daemon: No such container: 07a36908da510a31a214848c199da1a517bc5bb5476749a529936dccd839a
May 18 18:18:32 deis-1 docker[24371]: deis-logger
May 18 18:18:32 deis-1 systemd[1]: Starting deis-logger...
May 18 18:18:32 deis-1 systemd[1]: Started deis-logger.
May 18 18:18:32 deis-1 sh[24419]: Starting syslog...
May 18 18:18:32 deis-1 sh[24419]: Syslog server started...
May 18 18:18:32 deis-1 sh[24419]: deis-logger running

is everything that is there... The date of "Logs begin at", might coincide the date logging stopped working (although I am not sure).

When looking in the journactl of the machines actually running my apps, I can find ba ckthe logs...
Looks a bit like #3463 #3633 but I am not sure if actually the same...

Anything else I can do to debug the issue?

[bacongobbler]

How does the logs for logspout look? It doesn't look like the logger failed here (looks like normal operation to me)

[nathansamson]

deis-2 Full log (no cut off): http://pastebin.com/eLfipQhJ
deis-3 Partial log For some reason journalctl goes only back 3 days, possibly related to the log cutoff in the first place http://pastebin.com/emjMUTvm
deis-1 (the one running deis logger as well, similar cutoff as deis-3) http://pastebin.com/7praAQhf

Note at 18:13 I restarted (ps:restart, genius feature!!) my app, because at first I suspected my app (last log that I could see when looking up logs for my app was: nathan deployed vX).
At 18:20 I restarted logspout

[carmstrong]

@nathansamson Are there any reproduction steps you can provide? We don't see anything obvious in your logs that would cause issues, other than some fairly early etcd errors which prevented logspout from starting.

If you can replicate this or provide some steps for us to, it's something we'll certainly dig into. Thanks!

[mboersma]

Seems very likely #3854 is related to this. @nathansamson I think this is unlikely to happen with Deis v1.7.3.

[mboersma]

I'm going to close this because it seems very likely to be fixed by c73c8f7 in Deis v1.7.3. Please re-open if this issue persists on the current version.

[jorihardman]

@mboersma Just want to chime in that I've had this same issue on v1.7.3 as well as v1.9.0.

App logging stops completely and doesn't work until issuing a deisctl restart logspout && deisctl restart logger

[jorihardman]

Just wanted to include some additional info: My v1.9.0 cluster was made 2 days ago. It has one app with 6 instances and high volume log activity. App logging fails platform-wide, not just on one host.

[krancour]

@jorihardman see #3633

Can you see if the findings I presented there can be at all corroborated on your cluster?