listenerThreaded in Elasticsearch 5.0

[shivangshah]

Elasticsearch version: 5.0

Plugins installed: [x-pack]

JVM version: 1.8

OS version: Mac OS

Description of the problem including expected versus actual behavior: We were doing some performance test with Gatling and XPack transport client on localhost. The results were poles apart when the number of concurrent requests were increased (about 60 tps).

On profiling (visualVM) we found out that the bottle neck was BaseFuture$Sync.get(). On digging more I found this post here: #10766

I wanted to try out the exact same scenario with listenerThreaded(true) but seems like it is not available in Elasticsearch 5.0 transport client?

So, is that API still around to try? or is there a workaround?

[shivangshah]

so i see this being documented as breaking change in 2.x here: https://www.elastic.co/guide/en/elasticsearch/reference/2.3/breaking_20_java_api_changes.html#_automatically_thread_client_listeners

But we still run into the same problem as mentioned in #10766 where most of the time is actually spent on Sync.get(). Are there any particular configurations we need to look at in order to tune this ?

[shivangshah]

Also we are using PreBuiltXPackTransportClient as our client (because our cluster is secured with x-pack security)

[nik9000]

On profiling (visualVM) we found out that the bottle neck was BaseFuture$Sync.get(). On digging more I found this post here: #10766

Threads in that state are basically parked, waiting for work to do.

This isn't a bug or a feature request so I'm closing it. If you'd like to discuss it further head to discuss.elastic.co.

[shivangshah]

Sure @nik9000 .. Let me take this over to discuss.elastic.co. The problem is not them being parked .. the problem is transport client being the bottle neck of not being able to handle many concurrent requests. VisualVM profiler tells us that almost all the time, the threads are waiting on the sync.get() call.

Here's the gatling report.

The calls are going to one index and one shard (no replica). Everything is local as well.
As you can see that the initial calls returned back almost instantaneously, but over time requests started getting queued up and eventually when you monitor through a profiler, you can see all threads are waiting (blocked?) on Sync.get(). I hope this sheds any light. And if anyone can point me to any performance tests that were done using transport client, that'd be great as well.

[FS1360472174]

I am sad that met with this problem.
the throughtput is just 2/sec.
and cannot find way to solve it.