implement multi-line-timeout

[pzoleex]

well, it seems that syslog-ng does not detect any files in this mode and/or at least not displaying any incoming message.
To reproduce the issue: create the source file with this content:

root@thor-t410:/home/pzolee/zwa/projects/syslog-ng-ose-3.6/install# cat /var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt
Mar 18 12:07:04 testhost localprg PREFIX msg_with_prefix_without_garbage first line
msg_with_prefix_without_garbage second line
msg_with_prefix_without_garbage third line

then start syslog-ng with the config below.
The result:

[2014-06-18T09:43:28.842848] Module loaded and initialized successfully; module='syslogformat'
[2014-06-18T09:43:28.843001] Running application hooks; hook='1'
[2014-06-18T09:43:28.843019] Running application hooks; hook='3'
[2014-06-18T09:43:28.843054] syslog-ng starting up; version='3.6.0alpha0'

Thats all, syslog-ng does not recognize the file.

Expected result: syslog-ng processes the file then creates the proper multi line message when the timeout is reached:
"A message is considered complete if no new lines arrive to the message for 10 seconds, even if no line matching the multi-line-garbage() option is received."

config

@version: 3.6
options {
    threaded(yes);
};
source s_file_97358a54e6924af4826d9ab6ccb074fb {
    file("/var/testdb_working_dir/073358cb-d1ab-415a-a2a3-a874a500b080.txt" keep_hostname(yes)
 multi-line-mode(regexp)
 multi_line_prefix("PREFIX")
);
};
destination d_file_3f633b65908846c085561437c13d2ff8 {
    file("/var/testdb_working_dir/d333cb26-8a64-4bc1-8089-b04c1ace8a20.txt");
};
log {
source(s_file_97358a54e6924af4826d9ab6ccb074fb);
destination(d_file_3f633b65908846c085561437c13d2ff8);

flags(flow-control);
};

[talien]

Hi!

This is a docs bug, syslog-ng OSE never knew timeout with multi-line messages. It is an issue though, if there is no garbage arrives, syslog-ng would stop processing data from this source. But if you think it through, a missing garbage from the input means either the configuration is bad, or the input stream is bad. Garbage is mandatory to be able to process a whole message.

[pzoleex]

I don't agree, garbage is not required. In most cases, there is no garbage at all, for example tomcat log starts with date and that's all. A newer date indicates that a new message started.

Not speaking about possible message loss in a normal situation.

[talien]

You are right, I wasn't correct. What I wanted to mean is when no garbage or prefix is present for a long time, then it can be a misconfiguration. And what I wanted to demonstrate is that the lack of timeout in the OSE is not a bug, instead it is a missing feature. It can be implemented after the logproto refactor is merged into the OSE.

[faxm0dem]

just stumbled upon this, and I agree with @pzoleex it is a bug as the last message never makes it.
That being said, 10 seconds would be kind of arbitrary. If this is fixed, please make it configurable

[justcallmegreg]

It's a missing feature from syslog-ng OSE that has been never implemented. The docs says different though. This Note will be removed from the docs till the feature will be implemented. Try to use pre and suffix to ensure multi-line-messages to end.

An option could be implemented like multi-line-timeout(<N>) that closes the message if no new lines came in the last N secs.

[faxm0dem]

I stumbled again on this issue when trying to send some legacy multi-line logs from files with very low traffic. Many of those have one entry per file, and thus the log never makes it downstream.
It would at least make sense to have a SIGHUP "unstall" the event, but even that doesn't happen.

I'd love to have some idea as to if/when this will be implemented. thanks!

[phemmer]

This needs to be reclassified as a bug. Using multi-line-mode (with prefix or indent) will result in message loss.
If syslog-ng shuts down, or receives a SIGHUP, the last message received by a source with multi-line-mode is dropped.

Syslog-ng needs to flush the buffer when it:

• goes idle (configurable time period)
• shuts down
• reloads

[faxm0dem]

👍

[faxm0dem]

I'm bumping this issue, as one of our users complained again ;-)

[jasantiago-stratio]

It also misses the last message in case of a file rotation with multi-line-mode(indented), I think that a multi-line-timeout parameter, as @ngergo wrote, would be a very nice feature.