Logstash 2.2.x kafka output large increase in broker tcp connections

[thomasquatorze]

Upgrading from logstash 2.1.1 to 2.2.2 and seeing a rough 10x increase in established tcp connections to brokers. Running in a large environment this is a problem.

Testing env:

• ubuntu trusty - oracle java 1.7.0_51 - logstash 2.2.2
• one kafka output to one topic with 36 partitions on 6 brokers
• config using bootstrap_servers, topic_id & compression_type = snappy
• after a few mins logstash has 72 total established tcp connections to kafka brokers

Same config with logstash 2.1.1 maintain ~15 tcp connections
Setting "message_key" to multiple test strings (not starting with digit) had not effect.
Noticeable change between ls 2.1.1 & 2.2.2 is upgrade from jruby-kafka 1.4.0 to 1.5.0

[suyograo]

@thomasquatorze do you set a value for -w or --pipeline-workers ?

Noticeable change between ls 2.1.1 & 2.2.2

The bigger change from 2.1.x to 2.2.x is the re-architecture of pipeline which may cause this. Read about it https://www.elastic.co/blog/logstash-2-2-0-and-2-1-2-released and https://www.elastic.co/blog/upgrade-guide-for-logstash-2-2

[thomasquatorze]

No I did not set -w nor --pipeline-workers and workers in kafka output section was unset so it should have defaulted to 1. So in this case -w defaulted to 8 (number of cores in system) and created 8 pipelines each with its own output/connections ?

[ppf2]

We are seeing many producer instances on 2.x (same behavior for LS kafka output 0.8 and 0.9 beta), likely have the same underlying cause as this ticket.

[jordansissel]

logstash 2.1.1 to 2.2.2 and seeing a rough 10x increase

Logstash 2.2.0 introduced the new pipeline and defaulted the number of pipeline workers (and output workers) to the number of CPU cores available, so if you have 10 CPU cores (or close to it), then I would expect Logstash 2.2.x to have 10x the number of connections to Kafka as Logstash 2.1 did. We noticed this problem and reverted the output workers setting back to 1 by default in Logstash 2.3. If you are running Logstash 2.2, you can work around this by setting workers => 1 in any output you wish.

Given 2.2 is no longer receiving new features and 2.3 is available with this resolved (I believe), I will close this>