imptcp: sessions.openfailed counter

[amuhametov]

Expected behavior

no sessions.openfailed events should happen

Actual behavior

Getting multiple session.openfailed events

Fri Oct 12 15:13:43 2018: { "name": "imptcp(*\/514\/IPv4)", "origin": "imptcp", "submitted": 449680, "sessions.opened": 861, "sessions.openfailed": 668, "sessions.closed": 866, "bytes.received": 180257217, "bytes.decompressed": 0 }
Fri Oct 12 15:13:53 2018: { "name": "imptcp(*\/514\/IPv4)", "origin": "imptcp", "submitted": 682822, "sessions.opened": 868, "sessions.openfailed": 682, "sessions.closed": 869, "bytes.received": 271662777, "bytes.decompressed": 0 }
Fri Oct 12 15:14:03 2018: { "name": "imptcp(*\/514\/IPv4)", "origin": "imptcp", "submitted": 502525, "sessions.opened": 880, "sessions.openfailed": 718, "sessions.closed": 878, "bytes.received": 200937992, "bytes.decompressed": 0 }
Fri Oct 12 15:14:13 2018: { "name": "imptcp(*\/514\/IPv4)", "origin": "imptcp", "submitted": 213944, "sessions.opened": 867, "sessions.openfailed": 657, "sessions.closed": 861, "bytes.received": 87698028, "bytes.decompressed": 0 }

Steps to reproduce the behavior

Environment

• rsyslog version: 8.38.0
• platform: centos 7

impstats:

ResetCounters="on"

imptcp:

module(load="imptcp" threads="8" ProcessOnPoller="off")

input(
        type="imptcp"
        port="514"
        ruleset="ruleset 1"
        socketbacklog="65535"
        )

In my error log I have no messages like this:

"imptcp: connection could not be "
                                "established with host: %s"```

Could it be a of race between threads?

[jgerhards]

Could you explain the steps needed to reproduce the problem as it's a bit hard to understand at the moment? It would also be helpful if you could post your entire rsyslog.conf file and your debug log from the failed run.

[amuhametov]

I just start sending messages to my rsyslog :)
Here is the config

module(load="impstats"
       interval="10"
       severity="6"
       log.syslog="off"
       format="json"
       ResetCounters="on"
       log.file="/var/log/rsyslog_stats.log")


template(name="kafkamsg" type="string" string="%TIMESTAMP:::date-unixtimestamp% %HOSTNAME% %msg%")

module(load="imptcp" threads="8" ProcessOnPoller="off")
module(load="omkafka")

input(
        type="imptcp"
        port="514"
        ruleset="ruleset 1"
        socketbacklog="65535"
        )


ruleset(name="ruleset 1"
        queue.size="4000000"
        queue.type="LinkedList"
        queue.timeoutenqueue="60"
        queue.workerThreads="16"
        queue.dequeuebatchsize="50000"
        queue.timeoutWorkerthreadShutdown="600000"
        queue.timeoutshutdown="5000"
        queue.timeoutActionCompletion="5000"

        queue.saveOnShutdown="on"
        queue.maxFileSize="16g"
        queue.spoolDirectory="/var/spool/rsyslog"
        queue.filename="rsyslog_imptcp_input"
        queue.checkpointInterval="5"
        queue.dequeueSlowDown="20000"
        ) {

    if $syslogfacility-text == 'uucp' then {

        action(
                type="omkafka"
                topic="topic"
                partitions.auto="on"
                partitions.number="128"
                broker=#7 brokers
                confParam=["batch.num.messages=10000","message.max.bytes=20000000","queue.buffering.max.ms=1000","queue.buffering.max.messages=1000000","socket.timeout.ms=10000","broker.address.ttl=60000","message.send.max.retries=3","queue.buffering.max.kbytes=2097151"]
                errorFile="/var/log/rsyslog_kafka_errors"
                closeTimeout="4000"
                resubmitOnFailure="on"
                KeepFailedMessages="on"
                failedMsgFile="/var/spool/rsyslog/rsyslog_kafka_failed_msg"

                queue.type="LinkedList"
                queue.timeoutenqueue="60"
                queue.size="8000000"
                queue.dequeuebatchsize="50000"
                queue.workerthreads="16"
                queue.timeoutWorkerthreadShutdown="600000"
                queue.timeoutshutdown="6000"
                queue.timeoutActionCompletion="6000"

                queue.saveOnShutdown="on"
                queue.maxFileSize="16g"
                queue.spoolDirectory="/var/spool/rsyslog"
                queue.filename="rsyslog_action_omkafka"
                queue.checkpointInterval="5"
                queue.dequeueSlowDown="20000"

                action.resumeretrycount="-1"
                template="kafkamsg"
                action.errorFile="/var/log/rsyslog_kafka_action_error.log"
        )

        action(
                type="omfwd"
                Target="mirror_host"
                Port="514"
                Protocol="tcp"

                queue.type="LinkedList"
                queue.timeoutenqueue="60"
                queue.size="8000000"
                queue.dequeuebatchsize="50000"
                queue.workerthreads="4"
                queue.timeoutWorkerthreadShutdown="600000"
                queue.timeoutshutdown="6000"
                queue.timeoutActionCompletion="6000"

                queue.saveonshutdown="on"
                queue.maxFileSize="16g"
                queue.spoolDirectory="/var/spool/rsyslog"
                queue.filename="rsyslog_mirror"
                queue.checkpointInterval="5"
                queue.dequeueSlowDown="20000"

                action.resumeretrycount="-1"
                template="kafkamsg"
                action.errorFile="/var/log/rsyslog_mirror_action_error.log"
        )

        stop
    }
}

[rgerhards]

Is your problem that you see session.openfailed counters go up but no corresponding error messages? You use imptcp.

[amuhametov]

@rgerhards yes, thats it.

[amuhametov]

Found it. The problem caused by multiple senders that use BSD syslog format. All their messages were silently discarded. The only messages I had in logs are:
Framing Error in received TCP message from peer:...
received oversize message from peer...truncating...