Queue with omfwd discards all messages instead of writing to disk after second connection loss - rsyslogd has to be restarted to fix

[ed-mw]

Expected behavior

1. omfwd loses connection to the remote server.
2. Queue fills and starts writing to disk until maxDiskSpace is reached - then messages are discarded
3. Connection comes back online - all messages in disk queue are sent to remote server and disk size=0
4. omfwd loses connection to the remote server again... go to step 2.

Actual behavior

At step 4, the queue fills and then when it writes to disk, the messages are immediately discarded... and disk size=0.

Steps to reproduce the behavior

ruleset(name="graylogForwardAlt" queue.type="linkedList" queue.filename="graylogForwardAlt.queue"
queue.saveOnShutdown="on"
queue.maxDiskSpace="2m"
queue.checkpointInterval="10"
queue.size="200"
queue.timeoutEnqueue="0")
{
action(type="mmjsonparse")
action(type="omfwd" target="server.blah.blah" port="514"
template="jsonMsgTemplate"
protocol="tcp"
action.ResumeRetryCount="-1"
action.ResumeInterval="10"
)
}
module(
load="impstats"
interval="1"
resetCounters="on"
log.file="/var/log/rsyslog.stats"
format="json"
ruleset="graylogForwardAlt"
)

See attached stats log.

rsyslog-omfwd-bug-stats.log

The connection is lost and the size of the disk queue is increasing as messages are stored.
Once it fills up (size: 3697 in this test), messages get discarded as expected - starting at 15:30:35.
The connection is restored at 15:31:33 and the disk queue drains and is sent to the remote server.
Message processing proceeds normally.
The connection is lost again around 15:31:51 and the in-memory queue starts filling up.
After it fills up at 15:31:59 it gets sent to the disk queue - but immediately discarded... and then repeatedly after that.

If the connection is restored and lost again at this point, the behavior is the same.

The only way to get the disk queue working correctly again that I've found is to restart rsyslogd.

Environment

• rsyslog version:

rsyslogd 8.24.0-55.el7, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

• platform: CentOS Linux release 7.9.2009 (Core)