Documentation around how spooling works #2853
Comments
|
On Wed, 18 Jul 2018, tmahany419 wrote:
I am having issues where a host cannot reach a remote host to forward logs.
The logs are written to disk, but when the remote host is available these logs
are not always sent. I believe this is an issue with my configuration, however
I cannot find anything in the documentation about how spooling is supposed to
work for omfwd except for the action parameters ResumeRetry and
ResumeInterval.
Thanks for posting your config, I'm not seeing anything obviously wrong with it.
When the destination comes back up, new logs arriving have priority and are sent
first, when the current in-memory queue is low enough, rsyslog will read
messages in from the disk queue and process them. This happens far slower than I
think it should, but it's not something that anyone has tracked down yet.
The final disk queue file does not get deleted until rsyslog shuts down (to
avoid race conditions and overhead of trying to delete the file at the same time
new logs arrive that need to go in the file)
Please double-check that the logs are not getting delivered intermingled with
the new log messages
David Lang
… ```
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
# provides UDP syslog reception
module(load="imudp" threads="4" timeRequery="8" batchSize="128")
input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# Enable non-kernel facility klog messages
# $KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction off
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
#$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
#$IncludeConfig /etc/rsyslog.d/*.conf
# property(name="timestamp" dateFormat="rfc3339")
# constant(value=" ")
template(name="withip" type="list") {
constant(value="<")
property(name="pri")
constant(value="> ")
constant(value="10.100.100.253")
constant(value=" $$$ ")
property(name="fromhost-ip")
constant(value=" $$$ ")
property(name="rawmsg-after-pri")
constant(value="\n")
}
template(name="nochange" type="list") {
property(name="rawmsg")
}
# $IncludeConfig /etc/rsyslog.d/*.conf
action (
type="omfwd" target="192.168.254.12" port="10514" protocol="tcp"
RebindInterval="20000"
queue.size="50000"
queue.type="FixedArray"
queue.spoolDirectory="/var/spool/rsyslog-allsight"
queue.filename="myapp-spool"
queue.lowwatermark="48000"
queue.highwatermark="49000"
queue.discardmark="49999"
queue.maxfilesize="1g"
queue.saveonshutdown="on"
queue.dequeuebatchsize="100"
queue.fulldelaymark="49500"
action.ResumeInterval="1"
action.ResumeRetryCount="-1"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
template="withip"
)
```
### Environment
- rsyslog version: 8.31.0
- platform: Ubuntu 16.04 LTS
|
|
On Mon, 5 Nov 2018, Coleman McFarland wrote:
I have this question too. What is the behavior (default or configurable) if the target logger box (the one you ship to) is down?
If the target is down, rsyslog will try the configurable number of retries to
send it, and then throw the message away.
With TCP, the messages that rsyslog has submitted to the OS are considered
'sent', even if they aren't delivered due to system and/or network failures.
when the target comes back up, messages start flowing again.
If you have messages written to disk (disk assisted queues), then when the
memory queue fills up, messages are saved to disk, and sent intermingled with
the new messages when the target comes back up again.
David Lang
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am having issues where a host cannot reach a remote host to forward logs. The logs are written to disk, but when the remote host is available these logs are not always sent. I believe this is an issue with my configuration, however I cannot find anything in the documentation about how spooling is supposed to work for omfwd except for the action parameters ResumeRetry and ResumeInterval.
Environment
The text was updated successfully, but these errors were encountered: