Compress logs to use less disk space

[gnprice]

On the chat.zulip.org server, /var/log/zulip/ takes up a total of 4.8 GB of space (*). The bulk of that disk usage would end up looking the same on any other busy Zulip server with our default configuration. That's kind of a lot, and we can do better.

The space breaks down as (approximately)

• 1.5GB server.log and older versions
• 1.0GB django.log and older versions
• 1.0GB tornado.log and older versions
• 1.0GB other (some of which may not come from default Zulip behavior.)

The reason this is so much is

• For server.log, we have rotation, but we let the file grow past 500MB before rotating it, and we compress only the .2 and up. So two 500MB files, plus 9 more files (up to .10) which are compresed to about 50MB. This is configured in /etc/logrotate.d/zulip, which comes from puppet/zulip/files/logrotate/zulip. We should reduce the size we let the file grow to before rotation, and perhaps keep more files in order to keep the same total amount of log information.

• For django.log and tornado.log, we have rotation but no compression. We keep 10 old files for each, and they're 100MiB in size. This is configured through supervisord, from puppet/zulip/templates/supervisor/zulip.conf.template.erb. Unfortunately supervisor doesn't support log compression, which is the real solution for this. See Fixed and improved logrotate file #3385 and the supervisor issue Tim linked to from there. One solution would be to disable supervisor's log rotation, and use logrotate -- see a blog post -- though that post's solution involves using the copytruncate option, which isn't great. Following more breadcrumbs, there's this suggestion on a supervisor PR: use logrotate, and "send[] SIGUSR2 to Supervisor to signal it to reopen the logs." Maybe that'd work.

(*) Actually 5.3 GB, but at least 0.5GB are in some files it doesn't look like we currently generate, namely tornado.log*gz.

[zulipbot]

Hello @zulip/server-production members, this issue was labeled with the area: production label, so you may want to check it out!

[devZer0]

zulip apparently comes with logrotate.d configuration file - but it appears to be imcomplete as it only compresses some of the logs but not all relevant ones. at least django.log and tornado.log are quickly growing besides server.log

[timabbott]

that is by design, in that the log files managed by Supervisor (e.g. django.log and tornado.log) can't also use logrotate easily, see the discussion above. It'd be nice to improve this, but it's not trivial and could easily break things, so we'd need to carefully test one of the proposals mentioned above.

[timabbott]

We now have log rotation for all files that Zulip generates since 93d661d. So the only remaining issue is the supervisord thing. Sadly Supervisor/supervisor#322 remains unresolved.