Percolate failure

[paulkeogh]

Hi,

Firstly thanks for a great plugin.

I think there is a threading issue. When I run in Logstash with a worker count of 1, all is fine. When I run with a worker count of 4, I get the following error;

Failed to percolate event {:id=>nil, :target=>nil, :event=>#<LogStash::Event:0x445bf3f8 @metadata_accessors=#<LogStash::Util::Accessors:0x7d7b28ce @store={}, @lut={}>, @cancelled=false, @DaTa={"@timestamp"=>"2016-12-22T11:26:41.710Z", "@Version"=>"1", "tags"=>["multiline", "anonymous"], "type"=>"packet", "fingerprint"=>"e65fc68b9c85217fb92ff0be4c5dec189576b417", "Clear Text Message Payload"=>"%{User Data}"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x51cefd5 @store={"@timestamp"=>"2016-12-22T11:26:41.710Z", "@Version"=>"1", "tags"=>["multiline", "anonymous"], "type"=>"packet", "fingerprint"=>"e65fc68b9c85217fb92ff0be4c5dec189576b417", "Clear Text Message Payload"=>"%{User Data}"}, @lut={"tags"=>[{"@timestamp"=>"2016-12-22T11:26:41.710Z", "@Version"=>"1", "tags"=>["multiline", "anonymous"], "type"=>"packet", "fingerprint"=>"e65fc68b9c85217fb92ff0be4c5dec189576b417", "Clear Text Message Payload"=>"%{User Data}"}, "tags"], "[tags]"=>[{"@timestamp"=>"2016-12-22T11:26:41.710Z", "@Version"=>"1", "tags"=>["multiline", "anonymous"], "type"=>"packet", "fingerprint"=>"e65fc68b9c85217fb92ff0be4c5dec189576b417", "Clear Text Message Payload"=>"%{User Data}"}, "tags"], "[Clear Text Message Payload]"=>[{"@timestamp"=>"2016-12-22T11:26:41.710Z", "@Version"=>"1", "tags"=>["multiline", "anonymous"], "type"=>"packet", "fingerprint"=>"e65fc68b9c85217fb92ff0be4c5dec189576b417", "Clear Text Message Payload"=>"%{User Data}"}, "Clear Text Message Payload"]}>>, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>, :level=>:warn}

I am wondering if the underlying code is thread safe ?

[consulthys]

Thanks for reporting this. Which version of Logstash and ES are you using?

[paulkeogh]

Logstash 2.4 and elastic 2.3.3

I was running a load test on LS at ~ 400 tps when the error manifests itself.

[consulthys]

Ok, it seems more related to the underlying ruby client. It's conceptually the same as for the logstash-filter-elasticsearch plugin, which uses the same ruby client. I'll check if I can reproduce this.

[consulthys]

Also there's a good discussion about the thread safety of the underlying ruby client.