syslog input can not handle multi-line-mode

[lukasmrtvy]

syslog-ng

Version of syslog-ng

syslog-ng 3 (3.12.1)
Installer-Version: 3.12.1
Revision: 3.12.1-1
Compile-Date: Sep 22 2017 09:42:17
Module-Directory: /usr/lib/syslog-ng/3.12
Module-Path: /usr/lib/syslog-ng/3.12
Available-Modules: afuser,tfgetent,add-contextual-data,afstomp,kvformat,afsocket,csvparser,disk-buffer,afsql,cryptofuncs,mod-python,map-value-pairs,afamqp,cef,pseudofile,confgen,dbparser,pacctformat,afmongodb,mod-java,date,linux-kmsg-format,redis,graphite,afsmtp,riemann,syslogformat,snmptrapd-parser,afprog,xml,system-source,affile,sdjournal,geoip2-plugin,stardate,http,geoip-plugin,json-plugin,basicfuncs
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

Platform

Debian Jessie

Issue

Syslog input can not handle multi-line-mode

Failure

Error parsing afsocket, syntax error, unexpected LL_IDENTIFIER, expecting KW_NORMALIZE_HOSTNAMES or KW_USE_DNS or KW_USE_FQDN or KW_DNS_CACHE in /etc/syslog-ng/conf.d/custom.conf at line 5, column 100:
                                                                                                                                   included from /etc/syslog-ng/syslog-ng.conf line 162, column 1

Configuration

source s_stream { 
	syslog( ip(127.0.0.1) port(1999) transport("tcp") flags(no-parse) multi-line-mode(regexp)  multi-line-prefix(""^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}"")  ); 
};

[faxm0dem]

This is expected behaviour: multi-line support only works for file()

[lukasmrtvy]

DOCs:
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-syslog-chapter.html

So how can I parse multiline messages from syslog/socket/journald ? ( and forward it grouped as one message to another syslog? ) Impossible?

EDIT:
Im trying to do this: https://doc.logmatic.io/v2.0/discuss/58d4ee4ca113431b00133f27
... with syslog/socket/journald input.

[faxm0dem]

hrm indeed... @fekete-robert @bazsi doc-bug or bug?

[furiel]

syslog-source and multiline prefix seems a documentation bug for me. The syslog protocol over tcp works with syslog frames.
https://tools.ietf.org/html/rfc6587
3.4.1. Octet Counting
In this case the syslog frame specifies the end of the message, so using multiline option to specify the end of the message would conflict with the frame.

@muhahacz

• syslog
  syslog protocol does not terminate messages by endline, so multiline messages should work out of the box on them.

• sockets
  If you use any datagram based sockets, like unix datagram sockets or udp, then again, multiline messages will work out of the box, as in this case one datagram is one message.
  For stream based sockets, for now the open source edition does not support multiline option, neither for unix domain sockets nor tcp. For the unix domain part there is Multiline messages no longer handled by stream sockets. #1744.
  However in the premium edition version 6, there is multiline support for tcp and unix streams.

• journald
  I wrote a little python script that emits a multiline message, and the system source in syslog-ng read it as a single message. So multiline messages work out of the box on them.

For transmitting multiline messages: the easiest is if you use some transport that sends the message as a whole. For example any datagram protocol, or the syslog destination should work.

[gaborznagy]

The admin guide has been updated.
Thanks @muhahacz for discovering the doc bug.