rsyslog running at 100% while sending large syslog to unreachable host in omudpspoof

[iosappwithpaypal]

Expected behavior

omudpspoof should silently drop the message when it can't be forwarded to remote host. other actions of rsyslog like imfile, omfwd etc should continue to work.

Actual behavior

rsyslogd is at 100%. none of the syslogging works. other threads seem to be stuck. in strace of running rsyslog, we see that one thread is spinning in a loop.

Steps to reproduce the behavior

1. configure rsyslog to forward some messages using omudpspoof
2. send a message that is over default MTU of 1500.

Environment

• rsyslog version:
  8.29.0. also seen in 8.2001.0
• platform:
  CentOS-el7
• for configuration questions/issues, include rsyslog.conf and included config files

module( load="omudpspoof" )
template( name="EOS_ForwardFormat_info" type="list" ) {
constant( value="<160>" )
property( name="$year" )
constant( value=" " )
property( name="timestamp" dateFormat="rfc3164" )
constant( value=" nfc352.sjc.aristanetworks.com " )
property( name="syslogtag" position.from="1" position.to="32" )
property( name="msg" spifno1stsp="on" )
property( name="msg" )
}
$template EOS_ForwardSourceAddress_management_0,"fd7a:629f:52a4:4400::ac1e:8716"
$ActionOMUDPSpoofSourceNameTemplate EOS_ForwardSourceAddress_management_0

Redirect messages to (ns-management)2400:db08:face::1:514

$ActionOMUDPSpoofNetNsName ns-management
$ActionOMUDPSpoofTargetHost 2400:db08:face::1
$ActionOMUDPSpoofTargetPort 514
local4.=info;authpriv.!* :omudpspoof:;EOS_ForwardFormat_info

debugs from omudpspoof plugin while rsyslog is spinning:

6411.556152334:main Q:Reg/w0 : omudpspoof.c: inside omudpspoof.c:doAction
6411.556158073:main Q:Reg/w0 : omudpspoof.c: omudpspoof doTryResume entered, netns ns-management
6411.556169877:main Q:Reg/w0 : omudpspoof.c: 2400:db08:face::1:514/omudpspoof, src 'fdfd:5c41:712d:e037::af0:42ac', msg strt '<167>Apr 29 20:13:30 nfc317 ProcMgr-worker: %PROCMGR-7-PROCESSES_ADOPTED: ProcMgr (PID=2085) adopted running processes: (KernelNetworkInfo, PID=2688) (PhyEthtool, PID=2816) (Qos, PID=2691) (MicrosemiAgent, PID=2820) (McastCommon6, PID=2695) (CpuComplex, P'
6411.556175383:main Q:Reg/w0 : omudpspoof.c: UDPSend: pSockArray iS NULL...
6411.556180090:main Q:Reg/w0 : omudpspoof.c: omudpspoof doTryResume entered, netns ns-management
6411.556192330:main Q:Reg/w0 : omudpspoof.c: UDPSend: len 1685 maxpktlen 1480
6411.556197614:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 1: MF:1, hdrOffs 0, pktLen 1472
6411.556238583:main Q:Reg/w0 : omudpspoof.c: omudpspoof: write error (total len 1685): pktLen 1500, sent -1, fd 21: libnet_write_raw_ipv6(): -1 bytes written (Network is unreachable)

6411.556245275:main Q:Reg/w0 : omudpspoof.c: UDPSend: msgOffs 1472 pktLen 1472 len 1685
6411.556251553:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 2: MF:0, hdrOffs 1480, pktLen 213
6411.556261332:main Q:Reg/w0 : omudpspoof.c: omudpspoof: fragment write error len 1713, sent -1: libnet_pblock_coalesce(): packet assembly cannot find an IPv6 header

6411.556266762:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 2: MF:0, hdrOffs 1480, pktLen 213
6411.556273235:main Q:Reg/w0 : omudpspoof.c: omudpspoof: fragment write error len 1713, sent -1: libnet_pblock_coalesce(): packet assembly cannot find an IPv6 header

6411.556281863:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 2: MF:0, hdrOffs 1480, pktLen 213
6411.556288126:main Q:Reg/w0 : omudpspoof.c: omudpspoof: fragment write error len 1713, sent -1: libnet_pblock_coalesce(): packet assembly cannot find an IPv6 header

6411.556293197:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 2: MF:0, hdrOffs 1480, pktLen 213
6411.556299583:main Q:Reg/w0 : omudpspoof.c: omudpspoof: fragment write error len 1713, sent -1: libnet_pblock_coalesce(): packet assembly cannot find an IPv6 header

6411.556304892:main Q:Reg/w0 : omudpspoof.c: omudpspoof: stage 2: MF:0, hdrOffs 1480, pktLen 213
6411.556310991:main Q:Reg/w0 : omudpspoof.c: omudpspoof: fragment write error len 1713, sent -1: libnet_pblock_coalesce(): packet assembly cannot find an IPv6 header

[rgerhards]

@iosappwithpaypal can you please try the current daily stable build. It has a patch for omupdspoof, which could potentially address the issue.

• Ubuntu: https://www.rsyslog.com/downloads/download-daily-build/
• CentOS: https://www.rsyslog.com/rhelcentos-rpms/

[rgerhards]

mhhh... seeing this, I am not even sure if IPv6 is supported at all for this contributed module. I would hope that on a modern system this dirty hacky work-around is no longer needed.

@davidelang @alorbach Can you shed some light on this?

[davidelang]

I did not write it to support ipv6 David Lang

…

On Sat, 2 May 2020, Rainer Gerhards wrote: Date: Sat, 02 May 2020 04:12:23 -0700 From: Rainer Gerhards ***@***.***> Reply-To: rsyslog/rsyslog ***@***.***> To: rsyslog/rsyslog ***@***.***> Cc: David Lang ***@***.***>, Mention ***@***.***> Subject: Re: [rsyslog/rsyslog] rsyslog running at 100% while sending large syslog to unreachable host in omudpspoof (#4268) mhhh... seeing this, I am not even sure if IPv6 is supported at all for this contributed module. I would hope that on a modern system this dirty hacky work-around is no longer needed. @davidelang @alorbach Can you shed some light on this?

[iosappwithpaypal]

Thanks for taking a look. We (Arista networks) added Ipv6 support but the issue exists in ipv4 as well. The issue arises when sending a large syslog over the size of MTU to an unreachable host. When send fails for any of the fragments, it doesn't advance the offset for user payload. So it ends up in a loop there.

[rgerhards]

We (Arista networks) added Ipv6 support

can you contribute the patch?

[iosappwithpaypal]

Sure. Please let me know how to submit the patch.