Description
Our ThreadPool
constructor creates a couple of threads (scheduler and timer) which might not get shut down if the initialization of a node fails. A guice error might occur for example, which causes the InternalNode
constructor to throw an exception. In this case the two threads are left behind, which is not a big problem when running es standalone as the error will be intercepted and the jvm will be stopped as a whole. It can become more of a problem though when running es in embedded mode, as we'll end up with lingering threads.
The steps to reproduce are a bit exotic: create a tribe node, and make sure one of its inner client nodes initialization fails. Whether the threads are left behind or not depends on when the failure happens, whether it happens before or after the ThreadPool
has been initialized.
@Test
public void testThreadPoolLeakingThreadsWithTribeNode() {
Settings settings = ImmutableSettings.builder()
.put("node.name", "thread_pool_leaking_threads_tribe_node")
.put("tribe.t1.cluster.name", "non_existing_cluster")
//trigger initialization failure of one of the tribes (doesn't require starting the node)
.put("tribe.t1.plugin.mandatory", "non_existing").build();
try {
NodeBuilder.nodeBuilder().settings(settings).build();
} catch(Throwable t) {
//all good
assertThat(t.getMessage(), containsString("mandatory plugins [non_existing]"));
}
}
I think creating threads on the constructor is quite bad already, even worse since we haven't started the node yet. In general I would love to have more lightweight constructors, especially since we use guice. I think we should delay the threads creation to the actual start phase of the node, but this requires quite some changes as we currently schedule executions (threadPool.schedule
) from different objects constructors, which need the thread pool scheduler to be already initialized.