imfile: Rate limiting is not working

[dpkkumar01]

I have upgrade rsyslog 8.38 and tested the rate limiting option by configuring delay.message option in imfile. I am not getting the data at kafka end.

I have adjusted the delay between 1-1000. But it doesn't help.

Below is my configuration,

module(load="imfile" Mode="inotify")
module(load="mmnormalize")
module(load="omkafka")

template (name="MTFW_CDR" type="list") {
     constant(value="{\"ReportGeneratedTime\":\"")
     property(name="$!f2")
     constant(value=" +05:30")
     constant(value="\",\"Delivered Time\":\"")
     property(name="$!f3")
     constant(value=" +05:30")
     constant(value="\",\"Record Type\":\"")
     property(name="$!f5")
     constant(value="\",\"Delivery Status\":\"")
     property(name="$!f17")
     constant(value="\",\"Failure Reason\":\"")
     property(name="$!f18")
     constant(value="\",\"Failure Reason Type\":\"")
     property(name="$!f19")
     constant(value="\",\"Service Centre\":\"")
     property(name="$!f53")
     constant(value="\",\"Vendor\":\"NAME1\",\"Location\":\"IND\",\"IPAddress\":\"X.X.X.X\",\"Product\":\"MTFW\"}")
}

input(type="imfile"
     File="/home/rsyslog/cdrs/test*.log"
     Tag="cdr: "
     PersistStateInterval="100"
     deleteStateOnFileDelete="off"
     delay.message="100"
     #freshstartTail="on"
     ruleset="MTFW_CDR")

ruleset(name="MTFW_CDR") {
     action(type="mmnormalize" rulebase="/etc/MTFW_CDR.rb")
     action(
            action.resumeRetryCount="-1"
            action.resumeInterval="5"
            action.reportSuspension="on"
            action.reportSuspensionContinuation="on"
            broker=["X.X.X.X:9092"]
            template="MTFW_CDR"
            type="omkafka"
            topic="mtfw_cdr"
            partitions.auto="on"
            errorFile="/var/log/kafka_error.log"
            confParam=[
                       "socket.timeout.ms=1000",
                       "socket.keepalive.enable=true",
                       "queue.buffering.max.messages=50000",
                       "queue.buffering.max.ms=1000",
                       "compression.codec=gzip"
                       ]
                       queue.workerThreads="4"
                       queue.dequeueBatchSize="1000"
                       queue.highWatermark="1000000"
                       queue.lowWatermark="750000"
                       queue.spoolDirectory="/var/run/rsyslog/queues"
                       queue.fileName="stats_ruleset"
                       queue.maxDiskSpace="5g"
                       queue.size="5000000"
                       queue.saveOnShutdown="on"
           )stop
}

Please suggest.

[jgerhards]

Could you also provide a debug log?

[dpkkumar01]

Attached the debug logs for your reference.
debug_log.zip

[dpkkumar01]

@jgerhards , Did u checked the shared logs??
Do let me know, if you need any more information on the same.

Thanks in advance.

[jgerhards]

It seems everything is there. However, since im not familiar with the functionality, @rgerhards will have to take over.

[dpkkumar01]

@rgerhards , Can you please check and update the status on this case.

[rgerhards]

Inside the debug log, no file is read. If you think this is wrong, tell me a file name that is.

[dpkkumar01]

Please refer the attached debug logs,

debug_new.zip

Config details:

#GEMS-Rule-Name:MTFW_CDR-Start
module(load="imfile" Mode="inotify")
module(load="mmnormalize")
module(load="omkafka")

template (name="MTFW_CDR" type="list") {
     constant(value="{\"ReportGeneratedTime\":\"")
     property(name="$!f2")
     constant(value=" +05:30")
     constant(value="\",\"Delivered Time\":\"")
     property(name="$!f3")
     constant(value=" +05:30")
     constant(value="\",\"Record Type\":\"")
     property(name="$!f5")
     constant(value="\",\"Delivery Status\":\"")
     property(name="$!f17")
     constant(value="\",\"Failure Reason\":\"")
     property(name="$!f18")
     constant(value="\",\"Failure Reason Type\":\"")
     property(name="$!f19")
     constant(value="\",\"Service Centre\":\"")
     property(name="$!f53")
     constant(value="\",\"Vendor\":\"TELEDNA\",\"Location\":\"NOIDA\",\"IPAddress\":\"X.X.X.X\",\"Product\":\"MTFW\"}")
}

input(type="imfile"
     File="/home/prem/rsyslog/cdrs/DNA-AegisGw-CDR_PREPAID*.log"
     Tag="cdr: "
     PersistStateInterval="100"
     deleteStateOnFileDelete="off"
     #delay.message="100"
     #freshstartTail="on"
     ruleset="MTFW_CDR")

ruleset(name="MTFW_CDR") {
     action(type="mmnormalize" rulebase="/etc/MTFW_CDR.rb")
     action(
            action.resumeRetryCount="-1"
            action.resumeInterval="5"
            action.reportSuspension="on"
            action.reportSuspensionContinuation="on"
            broker=["X.X.X.X:9092"]
            template="MTFW_CDR"
            type="omkafka"
            topic="mtfw_cdr"
            partitions.auto="on"
            errorFile="/var/log/kafka_error.log"
            confParam=[
                       "socket.timeout.ms=1000",
                       "socket.keepalive.enable=true",
                       "queue.buffering.max.messages=50000",
                       "queue.buffering.max.ms=1000",
                       "compression.codec=gzip"
                       ]
                       queue.workerThreads="4"
                       queue.dequeueBatchSize="1000"
                       queue.highWatermark="1000000"
                       queue.lowWatermark="750000"
                       queue.spoolDirectory="/var/run/rsyslog/queues"
                       queue.fileName="stats_ruleset"
                       queue.maxDiskSpace="5g"
                       queue.size="5000000"
                       queue.saveOnShutdown="on"
           )stop
}

[rgerhards]

I don't see any read any further. Sure, this could be that it is stuck, but now crafted test and it works. Are you sure data is appended to the file? If in doubt, delete all state files and re-try.

[dpkkumar01]

I was done the same. I have removed all state file and restart rsyslog. I am unable to see any data transfer while enabling delay.message config.
After removing the delay.message config. Data started pushing to kafka.

[dpkkumar01]

@rgerhards , Do you need any other information to debug this issue??

[dpkkumar01]

@rgerhards , Kindly help me to close this case ASAP