Feature request: Have an option to compress log files after their rotation

[nilstoedtmann]

I know that there are tools like logrotate that can do rotate-and-compress [1]. But logrotate cannot enforce {streamname} _logfile _maxbytes just-in-time [2]. Therefore i preferred supervisor to handle the rotation.

---

[1] https://lists.supervisord.org/pipermail/supervisor-users/2010-November/000751.html
[2] I have processes that occasionally, e.g. if certain errors occur, fill logs rapidly. By the time logrotate runs next time, the limited storage on my embedded system is already full.

[arabus4chip]

+1

And even better: with configurable compress program and extension , like gzip/.gz, bzip2/.bz2, xz/.xz.

[DanMauger]

+1 - Would definitely like to see this

[vahdani]

+1

[fmela]

+1

[vltr]

+1 oh yes!

[mwf]

+1, it would be great!

[mpyrev]

+1

[danslimmon]

+1

[miliauskaite]

+1

[harai]

+1

[analytically]

+1

[elsmorian]

+1

[cbarraford]

+1

[cdesales]

+1

[xpzouying]

+1

[romainfun]

Note : Recent versions of logrotate include a maxsize option

[nilstoedtmann]

logrotate is not watching logs continuously. By the time logrotate gets around to check for [max]size, the logs might have already filled up the disk (as mentioned in the OP).

[fabrizioficci]

+1

[sunkencity]

How about having an option to replacing the rotation scheme ending number with a postfix of date_from-date_to?

/my_process.log
/my_process.2016-04-13_04.30.00-2016-04-14_07.59.59.log
/my_process.2016-04-14_08.00.00-2016-04-14_12.00.00.log

Pros:

• easy to know in which logfile to look for older data.
• easy to add cron job to compress
• easy to add cron job to archive
• each logfile has a unique name
• still easy to delete the oldest file
• no contention over logfiles being rotated while taking a backup
• can use supervisord without logrotate and thus not being prone to lose data while logs are rotated.

[sunkencity]

If that is a way forward that is acceptable as a patch. I'd be happy to implement it.

[keen99]

+1 - rather surprised (but the again, not considering issue with logfile ownership) that there's no option to compress rotated logs.

[noxdafox]

An implementation effort was made long ago #619 but it was not good enough.

Main issue is that Supervisor is implemented on top of an event-loop and file compression might take long time making Supervisor missing important events.

[EwaldvanGeffen]

-1: supervisor has it's features and logrotate has it's own. Let's not duplicate features, I would even prefer for supervisord to notify logrotate to start running.

[nilstoedtmann]

Well supervisor has already duplicated the main feature: rotation. This ticket is to add compression to the already existing rotation.

But in the end I don't really care whether the actual rotation and/or compression is done by supervisor itself, or by some other software. It is vital though that the rotation happens immediately when the logfile's size reaches its threshold.

So how would this look like? I guess supervisor should the not do any rotation (good luck having supervisor rotate, and logrotate only compress. I tried it, and it's a bad idea). There would be a new supervisor directive rotation_command = /usr/sbin/logrotate /etc/supervisor/logrotate.conf, and when *_logfile_maxbytes is hit, then instead of the internal rotation, the external rotation_command is executed. The logrotate.conf in question would have a size directive that is equal (or slightly smaller, to be on the safe side) than *_logfile_maxbytes. Finally there should be something like postrotate kill -USR2 $(cat /var/run/supervisord.pid) to tell supervisor to re-open its logfiles.

I have to admit this feels slightly cumbersome.

[vltr]

@EwaldvanGeffen I totally agree with you. Some time ago I voted a +1 in this same issue, but now I think that tools like supervisor should follow the "unix philosophy", plain and simple: do one thing and do it right; to be a part of the process and not the whole process. Logrotate does provide everything requested in here. Let supervisor do what it does best. Let logrotate do what it does best. For FAQ / RTFM sakes, it is feasible to provide an extra chapter or tutorial or how-to to connect both and that's it. Not only logrotator, but there are a couple of "cloud log tools" around that could be fed by supervisor - like Graphite, as an simple example.

The only point that matters here is what @nilstoedtmann stated:

But in the end I don't really care whether the actual rotation and/or compression is done by supervisor itself, or by some other software. It is vital though that the rotation happens immediately when the logfile's size reaches its threshold.

If supervisor guarantees that, I think this issue (and many others) would be satisfied. The idea to set your own rotation settings, also described by @nilstoedtmann, looks like an entry point for this, IMHO.

Well, that's just my 2 cents here 🙈

My best regards,
Richard.

[ioandev]

+1

[bezgubov]

+1

[rampmaster]

+1

[vikumar-ciena]

+1

[jonathanozvision]

+1

[timabbott]

Is it possible to configure logrotate to just do the compression part with supervisor doing the log rotation?

I don't see a clean way to do that, so it seems like the behavior most users actually want here is impossible with just combining supervisord as it stands and logrotate.

[esabol]

+1

[uda]

+1

[NickStallman]

+1

[cludokdrewicz]

+1

[d2-d2]

+1

[zacksleo]

+1

[mristin]

@vltr Could you please add to supervisord documentation how logrotator and supervisor could be coupled? Wouldn't there be problems with race conditions?

[vltr]

@mristin thanks for bringing my attention to this. I have used supervisor with logrotator some time ago (years) and I don't remember being caught on any race conditions; but I do remember setting some config parameters into logrotator to mitigate the possibility. I don't have any access to this configs anymore since I left the company I used this setup for quite some time now. Sorry for let you without a proper answer on this 😕

[roens]

Perhaps this would be a useful workaround for the lack of a compression option of rotated log files in supervisord?

[esabol]

Perhaps this would be a useful workaround for the lack of a compression option of rotated log files in supervisord?

Maybe my version of supervisord is old or something, but this would actually break supervisord’s log rotation in my experience.

[nilstoedtmann]

Perhaps this would be a useful workaround for the lack of a compression option of rotated log files in supervisord?

How does this behave when supervisor's rotation and the (potentially slow) cron compression overlap? I smell race conditions.

[mator]

can we have at least (simple) subprocess implementation first to fix the issue with compression of old log files, based on PR #619 (@noxdafox), instead of waiting indefinitely for "a proper" fix with eventlistener?

i believe, subprocess implementation would fix issue for 99.9% of people voted in this issue.

thanks

[rrauenza]

can we have at least (simple) subprocess implementation first to fix the issue with compression of old log files, based on PR #619 (@noxdafox), instead of waiting indefinitely for "a proper" fix with eventlistener?

i believe, subprocess implementation would fix issue for 99.9% of people voted in this issue.

I'm confused why that other proposed implementation uses tar to compress multiple (I didn't dig into the PR), but could this also be done asynchronously in threading[1] or even better multiprocessing?

[1] yeah, the GIL, but we'd be doing I/O, and maybe supervisord can handle the timeslice sharing anyway since it isn't CPU intensive?

[oldcai]

can we have at least (simple) subprocess implementation first to fix the issue with compression of old log files, based on PR #619 (@noxdafox), instead of waiting indefinitely for "a proper" fix with eventlistener?
i believe, subprocess implementation would fix issue for 99.9% of people voted in this issue.

I'm confused why that other proposed implementation uses tar to compress multiple (I didn't dig into the PR), but could this also be done asynchronously in threading[1] or even better multiprocessing?

[1] yeah, the GIL, but we'd be doing I/O, and maybe supervisord can handle the timeslice sharing anyway since it isn't CPU intensive?

I'll provide a perspective in the shoes of a mountainous person: we can kill a subprocess safely but not a thread.

Plus, this feature request had been there for almost a decade.

[oldcai]

In my opinion, if the log rotation function is not complete, it shouldn't be enabled by default and just depend on logrotate. This could decrease the confusion.

[mator]

Is someone (developer) brave enough to close this issue as 'DECLINED' , than keeping issue open for 10 years long?!? Just tell people that there's no way to compress logs, logrotate does not supported, in-application compression is not supported, resistance is futile. Thanks.

[timabbott]

I don't think closing issues for important problems in popular software just because they've proven difficult to solve and thus remain open for many years makes sense. The fact that an issue is open in no way promises an immediate solution, and I don't think anyone should be pressuring maintainers of the valuable project to close it as wontfix just because they're sad this feature isn't implemented.

…

-Tim Abbott (mobile)

On Fri, Nov 5, 2021, 04:54 Anatoly Pugachev ***@***.***> wrote: Is someone (developer) brave enough to close this issue as 'DECLINED' ? Just tell people that there's no way to compress logs, logrotate does not supported, in-application compression is not supported, resistance is futile. Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#322 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU6NWSTDUOVQJGFMGEA5ULUKPAXVANCNFSM4AJ7ERWQ> .