Deprecate multiline filter plugin in favor of multiline codec

[suyograo]

Multiline (ML) event processing is a popular use case in Logstash. Multiline event processing is complex and relies on proper event ordering. The best way to guarantee ordered log processing is to implement the processing as early in the pipeline as possible.

Today, users have 2 choices - multiline filter and multiline codec. The preferred tool in the Logstash pipeline is the multiline codec, which merges lines from a single input using a simple set of rules. This can be used with any input source.
Multiline filter performs a similar task of aggregating multiple lines, and exists because it predates the codec concept in Logstash. This filter is not thread-safe and was advised not to be used when setting filter_worker or -w to more than 1.

Recently, there has been a number of key enhancements in ML codec - proper handling of streams from multiple sources, auto-flushing of buffered lines, circuit breakers for preventing OOMs and so on. At this point we feel ML filter has run its course, and needs to be deprecated. This would also avoid confusion for new users - having to pick between the two.

The plan is to deprecate ML filter in version 2.2, and remove it from the default package in version 3.0. You can still install ML filter by using bin/plugin install, but our advice is to move to the more robust ML codec.

[runningman84]

Please do the multiline handling at filebeat. Right now filebeat might send a events to two different logstash servers which would prevent it from beeing processed as a mutline event.

[runningman84]

Please do the multiline handling at filebeat. Right now filebeat might send a events to two different logstash servers which would prevent it from beeing processed as a mutline event.

[suyograo]

@runningman84 absolutely. elastic/filebeat#89

[suyograo]

@runningman84 absolutely. elastic/filebeat#89

[mrjameshamilton]

How can I have different multiline patterns for different types of files? Currently, I have defined several multiline patterns and use ifs to choose between them based on the filename. Is this possible with the multiline codec? It seems that I can only define a single pattern.

[mrjameshamilton]

How can I have different multiline patterns for different types of files? Currently, I have defined several multiline patterns and use ifs to choose between them based on the filename. Is this possible with the multiline codec? It seems that I can only define a single pattern.

[rahulghanate]

I have the similar issue what @mrjameshamilton have mentioned.

[rahulghanate]

I have the similar issue what @mrjameshamilton have mentioned.