Log size limit settings?

[ineu]

Log files stored in /var/lib/deis/store/logs are not limited in size. I've recently had a problem with Ceph deadlock and while investigating, noticed that one of my log files was 18GB large. Looks like there are two improvements that can be made here:

1. There should be configuration option for logger to specify maximum number of log entries (or maximum file size), just like Heroku's 1500 lines limit. This is to avoid huge files on disk. Another option here is some kind of logrotate.
2. There should be configuration option or hardcoded value for maximum logs rate from an app, bytes/second. This is to avoid deadlocks in Ceph MDS when application with multiple containers on several hosts generates high amount of logs.

Please let me know if it makes sense.

[krancour]

A dupe of this issue was recently opened and breathed new life into this topic.

I think, at this point in time, we can safely rule out this issue being addressed. Development efforts are being concentrated on v2. v1.x is (mostly) receiving just bug fixes now. Also, the logger was massively overhauled just recently and with that overhaul came the possibility (which we highly recommend) of storing logs in memory. I'm going to elaborate on that because it's the detail that nearly renders this issue moot.

The logger component was never intended for long-term archiving of logs. It's a centralized point of aggregation where developers can query recent application logs via CLI -> controller -> logger. It's only useful for troubleshooting an issue in real time, and that's all it was ever meant for.

If one wants to aggregate and index logs for the sake of search, audit, analytics, error detection, etc. there are many existing solutions such as Papertrail, Loggly, and Splunk. Deis-logger doesn't want to compete with those products-- it would rather just feed logs to them.

This being the case, opting for in-memory storage of logs and "draining" those logs to an external archive is an optimal solution that can be achieved today-- and that strategy is likely to carry forward to v2, where file-based log storage might not even be an option.

Hopefully in-memory log storage + drain strikes operators as a reasonable option. If the community at large doesn't agree, we can re-open this at a later time.