flush-timeout() not implemented in elasticsearch2 destination

[sergio-limone]

syslog-ng

Version of syslog-ng

syslog-ng 3.9.1
Installer-Version: 3.9.1
Revision:
Module-Directory: //usr/lib64/syslog-ng
Module-Path: //usr/lib64/syslog-ng
Available-Modules: linux-kmsg-format,afstomp,confgen,dbparser,basicfuncs,afsocket,disk-buffer,date,sdjournal,cryptofuncs,afuser,graphite,afprog,pseudofile,csvparser,syslogformat,afamqp,affile,add-contextual-data,kvformat,json-plugin,cef,system-source
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on

Platform

CentOS 7
Linux saffa 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Issue

flush-timeout is not implemented for the elasticsearch2 destination.

Failure

When i have a flush-limit set, syslog-ng will wait forever (or till limit of messages is reached), before sending the logs to the ES server. The only way to send the messages in the buffer to ES is to stop (or probably SIGNAL) syslog-ng

Steps to reproduce

• set flush-limit
• send less messages
• wait, messages will not arrive

Configuration

Relevant part of my config:

destination d_elastic_local {
        elasticsearch2(
                index("my-logs-${YEAR}.${MONTH}.${DAY}")
                type("my-log")
                cluster("elasticsearch")
                template("$(format-json --scope nv-pairs --exclude 0*,1*,2*,3*,4*,5*,6*,7*,8*,9*,A*,B*,C*,D*,E*,F*,G*,H*,I*,J*,K*,L*,N*,M*,O*,P*,Q*,R*,S*,T*,U*,V*,W*,X*,Y*,Z* @timestamp=${ISODATE})")
                client-lib-dir("/usr/share/elasticsearch/lib")
                flush-limit(2000)
                client-mode("http")
                cluster-url("http://localhost:9200")
        );
};

[faxm0dem]

I can confirm this behaviour, and indeed it can be a problem on low-traffic instances.
That being said, I believe the default value of flush-mimit(5000) is definitely too large.
A sane default would be between 1 and 100 IMHO.