possible implication of increasing the MaxMessageSize to 64K

[meetme2meat]

We found today that our application logs are getting truncated on writing to rsyslog file upon reading the rsyslog documentation I understood that the default limit is 8K. Given we are hitting this limit we are possibly thinking of increasing the MaxMessageSize to 32 - 64K but the documentation highlights that it can cause some memory footprint.

https://www.rsyslog.com/doc/master/configuration/global/index.html

Well I just wanted to confirm whether it is safe to have that number for the MaxMessageSize

Environment.

• rsyslog version: rsyslogd: version 8.1901.0
• platform: Linux Debian 64 bit Intel.

[davidelang]

It's going to depend on how large your messages really are, how much ram you have, and how large your queues get. implement impstats to let you see the memory footprint and queue sizes, but remember that when things go wrong, the queues build up. David Lang

…

On Mon, 15 May 2023, Viren Negi wrote: We found today that our application logs are getting truncated on writing to rsyslog file upon reading the rsyslog documentation I understood that the default limit is 8K. Given we are hitting this limit we are possibly thinking of increasing the MaxMessageSize to 32 - 64K but the documentation highlights that it can cause some memory footprint. [https://www.rsyslog.com/doc/master/configuration/global/index.html ](https://www.rsyslog.com/doc/master/configuration/global/index.html ) Well I just wanted to confirm whether it is safe to have that number for the MaxMessageSize ### Environment. - rsyslog version: rsyslogd: version 8.1901.0 - platform: Linux Debian 64 bit Intel.

[meetme2meat]

thanks @davidelang I guess 64k would be enough for us we have 8GB Ram. But I wonder why does rsyslog still truncate

when I have set the $MaxMessageSize 64k in the rsyslog.conf file

#/etc/rsyslog.conf.d
#1.1
$MaxMessageSize 64k
module(load="imuxsock")
module(load="imklog")
module(load="imudp")
input(type="imudp" port="514")
#$KLogPermitNonKernelFacility on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
#$FileOwner syslog
#$FileGroup adm
#$FileCreateMode 0640
#$DirCreateMode 0755
#$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf

I still see the truncation in the final result.

[davidelang]

please start rsyslog with the -o /path/to/file option and post the resulting combined config file. David Lang

…

On Mon, 15 May 2023, Viren Negi wrote: Date: Mon, 15 May 2023 05:34:30 -0700 From: Viren Negi ***@***.***> Reply-To: rsyslog/rsyslog ***@***.***> To: rsyslog/rsyslog ***@***.***> Cc: David Lang ***@***.***>, Mention ***@***.***> Subject: Re: [rsyslog/rsyslog] possible implication of increasing the MaxMessageSize to 64K (Issue #5137) thanks @davidelang I guess 64k would be enough for us we have 8GB Ram. But I wonder why does rsyslog still truncate when I have set the `$MaxMessageSize 64k` in the rsyslog.conf file ``` #/etc/rsyslog.conf.d #1.1 $MaxMessageSize 64k module(load="imuxsock") module(load="imklog") module(load="imudp") input(type="imudp" port="514") #$KLogPermitNonKernelFacility on $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $RepeatedMsgReduction on #$FileOwner syslog #$FileGroup adm #$FileCreateMode 0640 #$DirCreateMode 0755 #$Umask 0022 #$PrivDropToUser syslog #$PrivDropToGroup syslog $WorkDirectory /var/spool/rsyslog $IncludeConfig /etc/rsyslog.d/*.conf ``` I still see the truncation in the final result.

[rgerhards]

If this is via UDP, you may need to change networking components. Rainer Sent from phone, thus brief. Viren Negi ***@***.***> schrieb am Mo., 15. Mai 2023, 14:34:

…

thanks @davidelang <https://github.com/davidelang> I guess 64k would be enough for us we have 8GB Ram. But I wonder why does rsyslog still truncate when I have set the $MaxMessageSize 64k in the rsyslog.conf file #/etc/rsyslog.conf.d #1.1 $MaxMessageSize 64k module(load="imuxsock") module(load="imklog") module(load="imudp") input(type="imudp" port="514") #$KLogPermitNonKernelFacility on $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $RepeatedMsgReduction on #$FileOwner syslog #$FileGroup adm #$FileCreateMode 0640 #$DirCreateMode 0755 #$Umask 0022 #$PrivDropToUser syslog #$PrivDropToGroup syslog $WorkDirectory /var/spool/rsyslog $IncludeConfig /etc/rsyslog.d/*.conf I still see the truncation in the final result. — Reply to this email directly, view it on GitHub <#5137 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALJ3C6SGKZNUY4ZC3ZIYPTXGIPFRANCNFSM6AAAAAAYB62WTU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[meetme2meat]

@rgerhards I'm plainly writing the logs to stdout, even If I enabled Tcp for rsyslog how may I suppose (sent log over tcp) to ryslogd as I don't have a TCP client?

[meetme2meat]

@rgerhards Ok so I just learned that systemd is the client here when I write to stdout and systemd forward the logs to rsyslog over the unix /dev/log still not sure how can I ask it to use TCP over unix and would unix not work?

[davidelang]

systend deliberatly cripples that interface to syslog (they ignore new RFCs and common practice and act as if nothing has changed on the syslog side since the '90s) use the imjournal module in rsyslog instead. David Lang On Mon, 15 May 2023, Viren Negi wrote:

…

@rgerhards Ok so I just learned that `systemd` is the client here when I write to stdout and systemd forward the logs to rsyslog over the unix `/dev/log` still not sure how can I ask it to use TCP over `unix` and would unix not work?

[meetme2meat]

@davidelang @rgerhards is there a provision to split the message in smaller size? I see splitOversizeMessage but I'm not sure how is it used?

rsyslog/tools/rsyslogd.c

Line 1025 in a71589b

splitOversizeMessage(smsg_t *const pMsg)

[rgerhards]

It only affects messages that get larger during rsyslog processing, not on the regular input side. Also, while it will split the message, the result will not be very satisfying. If journal has no option to process larger messages, and inject them properly into rsyslog, you need to bypass journal.

[meetme2meat]

@rgerhards @davidelang I have a question

I just removed the following entry from the target.service file

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=sbagent

and following rsyslog conf

$MaxMessageSize 64k
module(load="imuxsock")
module(load="imklog")
module(load="imudp")
input(type="imudp" port="514")
#$KLogPermitNonKernelFacility on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
#$FileOwner syslog
#$FileGroup adm
#$FileCreateMode 0640
#$DirCreateMode 0755
#$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf

Now I see the logs are getting split. A 1.5 MB log was split which is something I wanted, but to understand how that happened I'm writing this.

Thanks

[meetme2meat]

@rgerhards @davidelang I understood who is splitting the message default stdout is written to the system journal by systemd. And journald conf has an entry LINE_MAX which split the large message greater than default 48K.

Now the only piece of the puzzle I would like to understand is when no imjournal module is enabled how come rsyslog is observing the journal message?

[davidelang]

journald has a setting where it moves the /dev/log that ryslog listens to and creates it's own, and then outputs it's messages to the moved /dev/log that rsyslog is listening to. There is far less data delivered than is in journald, but the base old 1k-limit minimal data gets delivered by journald, look into it's settings for details. David Lang P.S. if you log messages with the template RSYSLOG_DebugFormat you see lots of details about the message, including what im* module received the message

…

On Sun, 21 May 2023, Viren Negi wrote: @rgerhards @davidelang I understood who is splitting the message default stdout is written to the system journal by systemd. And `journald` conf has an entry `LINE_MAX` which split the large message greater than default 48K. Now the only piece of the puzzle I would like to understand is when **no** `imjournal` **module is enabled how come rsyslog is observing the journal message**?

[rgerhards]

it looks like this is fully answered, so closing the issue.

[meetme2meat]

@rgerhards, @davidelang Thanks for all the help, for future seekers. I have increased the MAX_MESSAGE_SIZE to 64K

and let the journald do the splitting see this #5137 (comment)