Major performance issues

[shaharmor]

Hey,

When using this plugin we see a huge drop of events/s that logstash processes.

We believe its because the upload of the files are happening on the main plugin thread instead of a different one like on S3/ES plugins.

Any chance to get this implemented?

[shacharz]

Any updates?

[joshdvir]

+1

[jordansissel]

@joshdvir Please don't just +1 issues. It creates hundreds of emails (to all watching the Logstash project) and doesn't include actionable details -- Is your +1 reporting that you are experiencing the same problem? Is your +1 in support of the proposed cause? Needs more details :)

If you really feel the need to +1 an issue, please use Github's reactions feature.

[joshdvir]

@jordansissel sorry about that, I have a situation right now with a service that has 3000 rps that I need Logstash to parse, I see a big performance hit when using this plugin when I set the upload interval low.
The way I'm dealing with it right now is on a 4 core server I run 4 logstash processes each with 2 pipeline worker, so I have 4 processes doing upload of files.
I've set the max_file_size_kbytes to 50MB and uploader_interval_secs is at 20 sec.

Would be happy to hear any recommendations on how to have better performance.

[josephlewis42]

This should be fixed with #29 released as 3.2.0, the plugin now uses a pool of upload workers configurable with max_concurrent_uploads, set to 5 by default.