Feature request: new configuration: 'timeout_is_error'

[macdjord]

I request that a new configuration option be added, 'timeout_is_error' (type boolean, default true). This option would be such that, if it is set to true, if the flush_interval expires on a buffer, an error event is generated and the content of the buffer is output under the timeout_label label (as is the current behavior), but if it is false, the content of the buffer is emitted as if it had been completed and no error is generated.

I have a number of logs which I process using concat. In each case, I can tell when a new record is starting (because it starts with a timestamp), but have no reliable way of telling when one ends, except by waiting for the next log entry to start. As a result, I get lots of bogus timeout errors from concat. Dealing with this adds a lot of pointless, hard-to-maintain complexity and repetition to my Fluendtd configuration files. It would be much easier if I could just tell concat that no, those timeouts are not errors and it should not treat them as if they were.

[okkez]

I will consider this issue in next week.
(I will add new option or ...)

[macdjord]

This does not solve my use case. I do not want to wait forever for the next line. In some cases, it could be hours or days before the next line occurs. If 60s go by with no new lines, that's probably it - I want to output what is in the buffer without generating an error.

[okkez]

Use timeout_label configuration.

[macdjord]

That's what I do currently, but it adds a lot of pointless complexity to my config files and generates spurious timeout warning messages. All I want is to output my timeouts the same way other lines are output.

[LiberQuack]

+1

Some time ago I requested the same in #7

[okkez]

Hmm...

• Use timeout_label
  • 👍 Reduce warnings
  • 👎 Complex duplicated configuration
• Not using timeout_label
  • 👎 warnings
  • 👎 Complex duplicated configuration

You want:

• No warnings when timeout occurs
• Need timeout and flush buffer and emit normal event
• No complex duplicated configuration

Right?

[macdjord]

Yes. When the timeout happens, just emit whatever's in the buffer, as if a startline had come in.

[okkez]

It is impossible to implement such a thing.
It is the limitation of Fluentd.

[macdjord]

Er? You already emit the buffer content, directed to timeout_label. Is it really not possible to simply do the same, but direct it to the default label instead?

And it's definitely possible to not emit an error message - just make line 188 conditional.

[okkez]

It is impossible.
If just call router.emit, it skips remained filters defined after filter_concat.

You can set log_level per plugin.

<filter>
  @type concat
  log_level warn
  # ... snip
</filter>

[freemanh]

I have a way to work around this issue, but still not perfect:
multiline_start_regexp /[\d-]+ [\d:.]+ *ERROR/

so, only error stack trace is printed, the multiline plugin is activated. of course, this config suppose you print stack trace only when error happened.

but this is not perfect as well, because when an error printed, if there is no more preceding logs, this error may not be flushed

[naerymdan]

Is there really no way to simply close a multiline capture when timeout is reached and process like normal? A lot of situations (java logs & stacktrace) have no clear multiline end.

The other plugins from fluentd, like the memory buffer ones seems to be able to simply apply a flush interval to their processing, at which point all the collected stuff simply goes on to be processed normally.

[mindw]

Something like this from fluentd 0.12.20+:

multiline_flush_interval

The interval of flushing the buffer for multiline format. The default is disabled.

If you set multiline_flush_interval 5s, in_tail flushes buffered event after 5 seconds from last emit. This option is useful when you use format_firstline option. Since v0.12.20 or later.

[okkez]

Is there really no way to simply close a multiline capture when timeout is reached and process like normal?

There is no way.
This plugin is filter, so we cannot emit normal event. In fact, filter cannot emit normal event. It is difference from output plugin.

The other plugins from fluentd, like the memory buffer ones seems to be able to simply apply a flush interval to their processing, at which point all the collected stuff simply goes on to be processed normally.

Those plugins are output plugin. But this is filter plugin.
We don't use Fluentd's buffer mechanism in this plugin.

[mindw]

Got it. An unfortunate fluentd limitation.

[okkez]

I've got a solution for this issue, but it is not perfect and not implemented.

• Add new configuration soft_flush_interval, this counts timeout in main thread
• if Fluentd does not send events to this filter plugin:
  • soft_flush_interval: does not flush buffer
  • flush_interval: flush buffer and timeout error occur
• if Fluentd sends events but not the end of multline:
  • soft_flush_interval: flush buffer as normal events
  • flush_interval: flush buffer and timeout error occur (if soft_flush_interval > flush_interval: this means soft_flush_interval is disabled)
• This solution introduces complexity to this pluigin

Is this enough or not?

[vladm]

@okkez this would not really help my case as when output happens, it always starts with a start-line followed by some arbitrary number of lines. Then there can be no output for prolonged period of time and new event that starts with a start-line.

[mdamir]

A workaround this would be to use the relabel plugin to label in both cases i.e. in case when concat gives timeout and in case when it does not and then, finally putting all your output plugins under that label. Here is an example:

<filter **>
@type concat
key "message"
multiline_start_regexp "/^Start/"
flush_interval 5
timeout_label "@NORMAL"
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
<label @NORMAL>
<match **>
@type stdout
</match>
</label>

This will solve the duplicacy issue.

[okkez]

Thanks you for comments.
I will add this to README.md.

[macavity23]

@mdamir 's workaround works nicely! Thanks :)