Issue #1590 Multiline filter and additional worker threads do not work together.

[lmorfitt]

Simple documentation update to show this is a limitation with the filter.

#1590

[elasticsearch-release]

Can one of the admins verify this patch?

[colinsurprenant]

Thanks. LGTM.

[jordansissel]

To be honest, I'd rather just fix the multiline filter to be threadsafe, but we'll go with this patch for now.

[varh1i]

Hi @jordansissel,

Is there any estimated time for the fix of multiline filter to be threadsafe?

I asked already an alternative solution here: #1754 (comment)
but didn't get any response so added a SO question too: http://stackoverflow.com/questions/29072517/how-to-process-multilines-in-logstash-with-multiple-worker-threads

Thanks in advance,
Balazs

[lmorfitt]

@varh1i When we had this issue in our prod environment and was struggling with performance, we came up with the below solution.

Logstash receiver -> Redis -> Logstash indexers (3 instances per 4 core) -> Elasticsearch on the same box. Not ideal but worked for our performance needs.

Logstash indexers consume 1 core for the worker and then shared the remaining core for input / outputs.

cheers,
Luke

[varh1i]

Thank you @lmorfitt for the suggestion, I got a similar response from SO too, will give it a try :)

[synaq]

Hi There,

We have a new ELK environment which we love, the only thing making it lame at the moment is the multiline filter not being threadsafe - we have a big setup.

Any chance this can happen? :)

[davidturing]

Please fix this bug, we are using Kafka+Logstash, in order to leverage the partition capability, we may enable the logstash multiple worker threads functions.

At least, ES should provide a work around. Thanks!
David