worker IDLE, waiting for work

[amine250]

I have an issue with Rsyslog running in Docker. Its purpose is to send logs from an NFS to an ElasticSearch server.

Expected behavior

Continuously sending newer logs that are being written to the log file.

Actual behavior

Rsyslog only sends logs upon start-up, then stays idle even after log file update.

9333.682541119:main Q:Reg/w0  : wti.c: main Q:Reg/w0: worker IDLE, waiting for work.

Environment

• rsyslog version: 8.2002
• platform: centos7-based Docker image

my rsyslog.conf is :

module(load="imuxsock")
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$IncludeConfig /etc/rsyslog.d/*.conf

my included conf file is :

# file
module(load="imfile")
# parser
module(load="mmnormalize")
# sender
module(load="omelasticsearch")
$WorkDirectory /var/log/audit-trail-plugin/
input(type="imfile"
      File="/var/log/audit-trail-plugin/log.0"
      reopenOnTruncate="on"
      Tag="audit-trail:"
)

main_queue(
    queue.workerThreads="1"
    queue.dequeueBatchSize="1000"
    queue.size="10000"
)

action(type="mmnormalize"
    rulebase="/opt/rsyslog/audit-trail.rb"
)

template(name="esindex"
         type="list")
{
    constant(value="validators")
    constant(value="-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

template(name="accesstemplate" type="list"){
    constant(value="{")
    constant(outname="log_type" value="validation" format="jsonf")
    constant(value=",")
    property(name="$!all-json" position.from="2")
}

if $parsesuccess == "OK" then {
    action(
        type="omelasticsearch"
        server="elasticsearchserver.example.com"
        serverport="443"
        usehttps="on"
        template="accesstemplate"
        searchIndex="esindex"
        searchType="audit-trail"
        dynSearchIndex="on"
        bulkmode="on"
        maxbytes="100m"
        action.resumeretrycount="-1"
    )
}

Thanks for your help

[amine250]

Update: I have the same behaviour on a virtual machine, so it's not related to Docker.

The issue might be due to the NFS. Because of network delay, many lines are written at the same time to the logfile.

Any ideas, guys?

[davidelang]

so are you saying that /var/log/audit-trail-plugin/log.0 is on nfs? you may need to switch to polling mode to notice changes to the file instead of being able to rely on inotify. David Lang

…

On Wed, 15 Apr 2020, Amine wrote: Date: Wed, 15 Apr 2020 05:48:28 -0700 From: Amine ***@***.***> Reply-To: rsyslog/rsyslog ***@***.***> To: rsyslog/rsyslog ***@***.***> Cc: Subscribed ***@***.***> Subject: [rsyslog/rsyslog] worker IDLE, waiting for work (#4248) I have an issue with Rsyslog running in Docker. Its purpose is to send logs from an NFS to an ElasticSearch server. ### Expected behavior Continuously sending newer logs that are being written to the log file. ### Actual behavior Rsyslog only sends logs upon start-up, then **stays idle even after log file update.** ``` 9333.682541119:main Q:Reg/w0 : wti.c: main Q:Reg/w0: worker IDLE, waiting for work. ``` ### Environment - rsyslog version: 8.2002 - platform: centos7-based Docker image my **rsyslog.conf** is : ``` module(load="imuxsock") $WorkDirectory /var/lib/rsyslog # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $IncludeConfig /etc/rsyslog.d/*.conf ``` my included conf file is : ``` # file module(load="imfile") # parser module(load="mmnormalize") # sender module(load="omelasticsearch") $WorkDirectory /var/log/audit-trail-plugin/ input(type="imfile" File="/var/log/audit-trail-plugin/log.0" reopenOnTruncate="on" Tag="audit-trail:" ) main_queue( queue.workerThreads="1" queue.dequeueBatchSize="1000" queue.size="10000" ) action(type="mmnormalize" rulebase="/opt/rsyslog/audit-trail.rb" ) template(name="esindex" type="list") { constant(value="validators") constant(value="-") property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4") constant(value=".") property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7") constant(value=".") property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10") } template(name="accesstemplate" type="list"){ constant(value="{") constant(outname="log_type" value="validation" format="jsonf") constant(value=",") property(name="$!all-json" position.from="2") } if $parsesuccess == "OK" then { action( type="omelasticsearch" server="elasticsearchserver.example.com" serverport="443" usehttps="on" template="accesstemplate" searchIndex="esindex" searchType="audit-trail" dynSearchIndex="on" bulkmode="on" maxbytes="100m" action.resumeretrycount="-1" ) } ``` Thanks for your help

[amine250]

Hi @davidelang,
I will try it and keep you updated, thanks

[amine250]

So, I tried the polling mode, my imfile configuration looked like this

input(type="imfile"
      mode="polling"
      File="/var/log/audit-trail-plugin/log.0"
      reopenOnTruncate="on"
      Tag="audit-trail:"
)

But unfortunately it didn't work.

If it's worth noting, I'm using AWS EFS as a filesystem.

[d2avids]

So, I tried the polling mode, my imfile configuration looked like this

input(type="imfile"
      mode="polling"
      File="/var/log/audit-trail-plugin/log.0"
      reopenOnTruncate="on"
      Tag="audit-trail:"
)

But unfortunately it didn't work.

If it's worth noting, I'm using AWS EFS as a filesystem.

Did you fix that problem? Nothing helped for me neither