Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compressed dynafile output corruption #933

Open
davidelang opened this issue Mar 31, 2016 · 12 comments

Comments

Projects
None yet
8 participants
@davidelang
Copy link
Contributor

commented Mar 31, 2016

I have the template

$template perserverjson,"/var/log/servers/%$year%/%$month%/%$day%/%hostname:::lowercase%/%hostname:::lowercase%.%$year%%$month%%$day%.%$hour%-%$qhour%.json.gz"

and the action:

action(type="omfile" template="json" dynafile="perserverjson" dynafilecachesize="1200" ziplevel="9" flushinterval="60" name="perserverjson" ioBufferSize="512k" )

but if I go into this directory and try to access the zip files, I get errors.

zgrep -c ' ' _.17_json.gz

gzip: las1prsvc001.20160330.17-00.json.gz: invalid compressed data--format violated
las1prsvc001.20160330.17-00.json.gz:74

gzip: las1prsvc001.20160330.17-01.json.gz: invalid compressed data--format violated
las1prsvc001.20160330.17-01.json.gz:0

gzip: las1prsvc001.20160330.17-02.json.gz: invalid compressed data--format violated
las1prsvc001.20160330.17-02.json.gz:0

gzip: las1prsvc001.20160330.17-03.json.gz: invalid compressed data--format violated
las1prsvc001.20160330.17-03.json.gz:0

ls -l _.17_json.gz
-rw-r--r-- 1 root root 24442 Mar 30 17:14 las1prsvc001.20160330.17-00.json.gz
-rw-r--r-- 1 root root 27403 Mar 30 17:30 las1prsvc001.20160330.17-01.json.gz
-rw-r--r-- 1 root root 32554 Mar 30 17:45 las1prsvc001.20160330.17-02.json.gz
-rw-r--r-- 1 root root 32656 Mar 30 18:00 las1prsvc001.20160330.17-03.json.gz

@davidelang davidelang added the bug label Mar 31, 2016

@davidelang

This comment has been minimized.

Copy link
Contributor Author

commented Mar 31, 2016

This is with 8.18 pre git tree as of 2016-03-30, but it's not a new error, I've been getting some erros with these files for many versions.

@rgerhards

This comment has been minimized.

Copy link
Member

commented Mar 31, 2016

Just to make sure: you cannot access currently open files. Might this be
the Problem?

Sent from phone, thus brief.
Am 31.03.2016 03:24 schrieb "davidelang" notifications@github.com:

This is with 8.18 pre git tree as of 2016-03-30, but it's not a new error,
I've been getting some erros with these files for many versions.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#933 (comment)

@davidelang

This comment has been minimized.

Copy link
Contributor Author

commented Mar 31, 2016

no, it happens with older files as well. Things many HUPs old.

David Lang

On Thu, 31 Mar 2016, Rainer Gerhards wrote:

Just to make sure: you cannot access currently open files. Might this be
the Problem?

Sent from phone, thus brief.
Am 31.03.2016 03:24 schrieb "davidelang" notifications@github.com:

This is with 8.18 pre git tree as of 2016-03-30, but it's not a new error,
I've been getting some erros with these files for many versions.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#933 (comment)


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#933 (comment)

@Omniflux

This comment has been minimized.

Copy link

commented Oct 11, 2016

I see this too. My logrotate issues "invoke-rc.d rsyslog rotate" which runs "start-stop-daemon --stop --signal HUP --quiet --pidfile $PIDFILE --exec $DAEMON" at 6:25am.

All compressed log files are corrupt and end at this time and data is lost until a new log file is created and HUP issued, OR dynafile template causes a new file to be created.

Version: (Debian) 8.16.0-1~bpo8+1

template (name="DailyPerHostLogs"
type="list") {
constant (value="/var/log/remote/")
property (name="hostname" securepath="replace")
constant (value="/")
property (name="$year")
constant (value="/")
property (name="$month")
constant (value="/")
property (name="hostname" securepath="replace")
constant (value=".")
property (name="$now")
constant (value=".log.gz")
}

   action (type="omfile"
            dynafile="DailyPerHostLogs"
            dynaFileCacheSize="100"
            zipLevel="9"
            flushOnTXEnd="off"
            ioBufferSize="128k"
            asyncWriting="on")
@zhxiaom5

This comment has been minimized.

Copy link

commented Dec 10, 2016

this also happend to me,after i restart the rsyslogd,i zcat the gzip file ,same error appeared. it seems the omfile not properly handle the gzip file,when rsyslog receive signal

@Omniflux

This comment has been minimized.

Copy link

commented Jul 5, 2017

Still occurs with Debian version 8.24.0-1.

Is there any more information I can give to help resolve this?

@kortemik

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2017

+1 i have experienced this bug as well, i hope to provide example configuration that causes it as soon as possible

@StrongestNumber9

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2017

template (name="DailyPerHostLogs" type="list") {
         constant (value="/var/log/remote/")
         property (name="hostname" securepath="replace")
         constant (value="/")
         property (name="$year")
         constant (value="/")
         property (name="$month")
         constant (value="/")
         property (name="hostname" securepath="replace")
         constant (value=".")
         property (name="$now")
         constant (value=".log.gz")
}

action (type="omfile"
       dynafile="DailyPerHostLogs"
       dynaFileCacheSize="100"
       zipLevel="9"
       flushOnTXEnd="off"
       ioBufferSize="128k"
       asyncWriting="on"
)

action (type="omfile"
       dynafile="DailyPerHostLogs"
       dynaFileCacheSize="100"
       zipLevel="9"
       flushOnTXEnd="off"
       ioBufferSize="128k"
       asyncWriting="on"
)

input
(
        type="imfile"
        File="/tmp/test_omfile.log"
        Tag="test_omfile"
        Severity="info"
        Facility="local0"
        PersistStateInterval="1"
)

Simply having two omfile directives that write to same file will break it.

echo "Hello" >> /tmp/test_omfile.log
# zcat test.domain.tld/2017/08/test.domain.tld.2017-08-21.log.gz

gzip: test.domain.tld/2017/08/test.domain.tld.2017-08-21.log.gz: invalid compressed data--format violated
@liveder

This comment has been minimized.

Copy link

commented May 14, 2019

Since it's impossible to append GZ file after receiving HUP signal there is an option to solve this issue with other way.
If we will append to DYNFILE template additional variable something like %RSYSLOG_PROCID%(not implemented) or %RSYSLOG_STARTTIME%(not implemented) then it will be possible to split files into several each time rsyslog restarts. So GZ file won't be corrupted.

@ondrejkopka

This comment has been minimized.

Copy link

commented Jun 12, 2019

+1 exactly same issue here :(

@liveder

This comment has been minimized.

Copy link

commented Jun 12, 2019

there is a workaround using veryRobustZip.
it will provide some extra overhead but it works great

@rgerhards

This comment has been minimized.

Copy link
Member

commented Jun 12, 2019

I revisited this issue, and I am still unable to reproduce. @ondrejkopka could you provide a debug log of a session where this problem occurs?

@rgerhards rgerhards added potential bug and removed bug labels Jun 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.