Rsyslog imfile + omkafka truncating events, $MaxMessageSize not tanking effect

[ESMaletMa]

Hi All,

I'm tried using: rsyslogd 8.1911.0 (aka 2019.11) and 8.32.0

We have truncated events in the destination (maximum event I have is 5.6K), the end of the event is truncated. The destination is Azure Event Hubs (Kafka). Destination confirmed that events are already arriving truncated... I have:

#Set default maximum message size
$MaxMessageSize 512k

My config:

input(
type="imfile"
File="/datadisk/disk1/incapsula/mega_big26gb.raw"
Tag="akamai"
ruleset="publish_eh_akamai"
)

ruleset(name="publish_eh_akamai"){
action(
name="socdap-incapsula-4p-5m-rsyslog"
partitions.auto="on"
action.resumeRetryCount="-1"
action.resumeInterval="2"
broker=["dev-01.servicebus.windows.net:9093"]
type="omkafka"
template="rawmessage"
confParam=["linger.ms=500", "security.protocol=SASL_SSL", "sasl.mechanisms=PLAIN", "sasl.username=$ConnectionString", "sasl.password=Endpoint=.....="]
resubmitOnFailure="on"
keepFailedMessages="on"
failedMsgFile="/opt/lib/rsyslog_producer/producer_failed.log"
topic="stress_test"

    queue.type="LinkedList"
    queue.dequeueBatchSize="400000"
    queue.highWatermark="700000000"
    queue.lowWatermark="250000"
    queue.spoolDirectory="/datadisk/rsyslog"
    queue.filename="socdap-akamai-nespresso.queue"
    queue.maxDiskSpace="150g"
    queue.size="1000000000"
    queue.saveonshutdown="on"

    )
    stop

}

Any idea is welcome…

Thank you very much

[ESMaletMa]

I tried this parameter but nothing.... the same result: discardTruncatedMsg="on"

input(
type="imfile"
discardTruncatedMsg="on"
File="/datadisk/disk1/incapsula/mega_big26gb.raw"
Tag="akamai"
ruleset="publish_eh_akamai"
)

[davidelang]

you have resubmitonfailure turned on, in the kafka config. So something is happening where an error is returned to rsyslog while kafka is actually processing the message. I don't know what debugging you can enable at the kafka level, but I'd bet that the problem you are running into is happening there. Since this only happens at very high volumes of traffic, I'd guess that some queue there is filling up. try enabling impstats so you can see the details at the rsyslog level of how many messages rsyslog thinks it's submitting (hopefully it will tell you how many it's getting failures for as well) David Lang On Tue, 10 Mar 2020, ESMaletMa wrote:

…

Date: Tue, 10 Mar 2020 05:58:39 -0700 From: ESMaletMa ***@***.***> Reply-To: rsyslog/rsyslog ***@***.***> To: rsyslog/rsyslog ***@***.***> Cc: Subscribed ***@***.***> Subject: [rsyslog/rsyslog] Rsyslog imfile + omkafka duplicating events (#4214) Hi All, I'm using: rsyslogd 8.1911.0 (aka 2019.11) Rsyslog Configuration below. With that configuration in my final destination I have much more events than I should have. Something is duplicating events. If I send around 400.000 events everything goes well, I get 400.000 in my final destination. If I send, for instance 6 milions of events it goes crazy, in the final destination I get 40 milion… If I do a "deduplication" of those 40 milion I get, more or less 6 milion… so it is clear we are duplicating events. In my Kafka, which is in real an Azure Event Hubs I have 24 partitions and the throughtput is configured to the maximum available. I have tried to configure the timeout, the acks to try to avoid this behaviour. But I'm running out ideas... In Rsyslog logs I get some errors regarding the queue, it gets full, below also errors I see: input( type="imfile" File="/data/disk1/toDatalake/data_raw/*.raw" Tag="akamai" ruleset="publish_eh_akamai" ) ruleset(name="publish_eh_akamai"){ action( name="publish_to_socdap-akamai-lds" partitions.auto="on" action.resumeRetryCount="-1" action.resumeInterval="2" broker=["digital-eventhubs-dev-01.servicebus.windows.net:9093"] type="omkafka" template="rawmessage" confParam=["linger.ms=100", "security.protocol=SASL_SSL", "sasl.mechanisms=PLAIN", "sasl.username=$ConnectionString", "sasl.password=Endpoint=sb://digital-eventhubs-dev-01.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=cDokUkEsG9QeTBpIGTfo1ffHBoR2vG9Q/4uzQESuWQs=", "request.timeout.ms=15000"] resubmitOnFailure="on" keepFailedMessages="on" closeTimeout="15000" failedMsgFile="/opt/lib/rsyslog_producer/producer_failed.log" topic="socdap-akamai-lds-nespresso-25p" queue.type="LinkedList" queue.dequeueBatchSize="40000" queue.highWatermark="450000" queue.lowWatermark="250000" queue.filename="socdap-akamai-nespresso.queue" queue.maxDiskSpace="1g" queue.size="900000" queue.saveonshutdown="on" ) stop } Errors: rsyslogd: action 'publish_to_socdap-akamai-lds-nespresso' suspended (module 'omkafka'), rsyslogd: action 'action-2-builtin:omfile' resumed (module 'builtin:omfile') [ omkafka: Failed to produce to topic 'socdap-akamai-lds-nespresso-25p' (rd_kafka_producev)partition -1: -184/Local: Queue full Any idea is welcome… Thank you very much -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #4214

[ESMaletMa]

Thank you. Exactly yes, it is happening when I have a high volume of data, for small volume there is no problem.