Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SimpleLocalTransportTests#testConcurrentSendRespondAndDisconnect failure #23942

Closed
martijnvg opened this issue Apr 6, 2017 · 1 comment
Closed

SimpleLocalTransportTests#testConcurrentSendRespondAndDisconnect failure #23942

martijnvg opened this issue Apr 6, 2017 · 1 comment
Assignees
@s1monw
Labels
:Distributed/Network Http and internode communication implementations >test Issues or PRs that are addressing/adding tests

Comments

@martijnvg
Copy link
Member

Fails because of a suite timeout and doesn't reproduce.

Build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-unix-compatibility/os=fedora/104

> Assumption #1: only tcp transport has a handshake method
  2> WARNING: Suite execution timed out: org.elasticsearch.transport.local.SimpleLocalTransportTests
  2> ==== jstack at approximately timeout time ====
  1> [2017-04-06T13:59:06,671][INFO ][o.e.t.l.SimpleLocalTransportTests] [testNotifyOnShutdown]: before test
  2> "elasticsearch[TS_B_9][local_transport][T#1]" ID=1084 WAITING on java.util.concurrent.LinkedTransferQueue@4aaecf9d
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@4aaecf9d
  1> [2017-04-06T13:59:06,680][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[71]}, bound_addresses {local[71]}
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  1> [2017-04-06T13:59:06,681][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[72]}, bound_addresses {local[72]}
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  1> [2017-04-06T13:59:06,683][INFO ][o.e.t.l.SimpleLocalTransportTests] Stop ServiceB now
  1> [2017-04-06T13:59:06,685][INFO ][o.e.t.l.SimpleLocalTransportTests] [testNotifyOnShutdown]: after test
  1> [2017-04-06T13:59:06,686][INFO ][o.e.t.l.SimpleLocalTransportTests] [testResponseHeadersArePreserved]: before test
  1> [2017-04-06T13:59:06,698][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[73]}, bound_addresses {local[73]}
  1> [2017-04-06T13:59:06,700][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[74]}, bound_addresses {local[74]}
  1> [2017-04-06T13:59:06,724][INFO ][o.e.t.l.SimpleLocalTransportTests] [testResponseHeadersArePreserved]: after test
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  1> [2017-04-06T13:59:06,726][INFO ][o.e.t.l.SimpleLocalTransportTests] [testSendRandomRequests]: before test
  2> "elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#13]" ID=1083 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#12]" ID=1080 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[TS_A][local_transport][T#4]" ID=1079 WAITING on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[TS_A][local_transport][T#3]" ID=1076 WAITING on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@7bae44fc
  1> [2017-04-06T13:59:06,729][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[75]}, bound_addresses {local[75]}
  1> [2017-04-06T13:59:06,730][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[76]}, bound_addresses {local[76]}
  1> [2017-04-06T13:59:06,731][INFO ][o.e.t.t.MockTransportService] publish_address {local[77]}, bound_addresses {local[77]}
  1> [2017-04-06T13:59:07,040][INFO ][o.e.t.l.SimpleLocalTransportTests] [testSendRandomRequests]: after test
  1> [2017-04-06T13:59:07,042][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTimeoutSendExceptionWithDelayedResponse]: before test
  1> [2017-04-06T13:59:07,044][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[78]}, bound_addresses {local[78]}
  1> [2017-04-06T13:59:07,045][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[79]}, bound_addresses {local[79]}
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  1> [2017-04-06T13:59:07,161][WARN ][o.e.t.t.MockTransportService] [TS_B] Received response for a request that has timed out, sent [113ms] ago, timed out [13ms] ago, action [sayHelloTimeoutDelayedResponse], node [{TS_A}{TS_A}{1NyEEx-aT3qaawS4qVn9dg}{local}{local[78]}], id [1]
  1> [2017-04-06T13:59:07,162][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTimeoutSendExceptionWithDelayedResponse]: after test
  1> [2017-04-06T13:59:07,163][INFO ][o.e.t.l.SimpleLocalTransportTests] [testVersionFrom0to0]: before test
  1> [2017-04-06T13:59:07,165][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[80]}, bound_addresses {local[80]}
  1> [2017-04-06T13:59:07,166][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[81]}, bound_addresses {local[81]}
  1> [2017-04-06T13:59:07,167][INFO ][o.e.t.l.SimpleLocalTransportTests] [testVersionFrom0to0]: after test
  1> [2017-04-06T13:59:07,168][INFO ][o.e.t.l.SimpleLocalTransportTests] [testBlockingIncomingRequests]: before test
  1> [2017-04-06T13:59:07,170][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[82]}, bound_addresses {local[82]}
  1> [2017-04-06T13:59:07,171][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[83]}, bound_addresses {local[83]}
  1> [2017-04-06T13:59:07,172][INFO ][o.e.t.t.MockTransportService] [TS_TEST] publish_address {local[84]}, bound_addresses {local[84]}
  1> [2017-04-06T13:59:07,173][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[85]}, bound_addresses {local[85]}
  1> [2017-04-06T13:59:07,175][INFO ][o.e.t.l.SimpleLocalTransportTests] [testBlockingIncomingRequests]: after test
  1> [2017-04-06T13:59:07,176][INFO ][o.e.t.l.SimpleLocalTransportTests] [testErrorMessage]: before test
  1> [2017-04-06T13:59:07,177][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[86]}, bound_addresses {local[86]}
  1> [2017-04-06T13:59:07,178][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[87]}, bound_addresses {local[87]}
  1> [2017-04-06T13:59:07,182][INFO ][o.e.t.l.SimpleLocalTransportTests] [testErrorMessage]: after test
  1> [2017-04-06T13:59:07,183][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTracerLog]: before test
  1> [2017-04-06T13:59:07,184][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[88]}, bound_addresses {local[88]}
  1> [2017-04-06T13:59:07,185][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[89]}, bound_addresses {local[89]}
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[TS_A][local_transport][T#2]" ID=1075 WAITING on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[TS_A][local_transport][T#1]" ID=1073 WAITING on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@7bae44fc
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  1> [2017-04-06T13:59:07,192][INFO ][o.e.c.s.ClusterSettings  ] updating [transport.tracer.include] from [[]] to [["test"]]
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  1> [2017-04-06T13:59:07,192][INFO ][o.e.c.s.ClusterSettings  ] updating [transport.tracer.exclude] from [["internal:discovery/zen/fd*","cluster:monitor/nodes/liveness"]] to [["DOESN'T_MATCH"]]
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#9]" ID=1070 WAITING on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@166fd7a5
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@166fd7a5
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2> 	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:248)
  2> 	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:91)
  2> 	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
  2> 	at org.elasticsearch.transport.AbstractSimpleTransportTestCase$13.doRun(AbstractSimpleTransportTestCase.java:596)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
  2> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@3f16dced
  2> "elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#4]" ID=1065 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6e85027f
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][[timer]]" ID=1061 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)
  2> "TEST-SimpleLocalTransportTests.testConcurrentSendRespondAndDisconnect-seed#[290BC1B07436B39B]" ID=858 WAITING on java.util.concurrent.CountDownLatch$Sync@2a8130d2
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@2a8130d2
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2> 	at org.elasticsearch.transport.AbstractSimpleTransportTestCase.testConcurrentSendRespondAndDisconnect(AbstractSimpleTransportTestCase.java:635)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  1> [2017-04-06T13:59:07,196][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTracerLog]: after test
  1> [2017-04-06T13:59:07,198][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTcpHandshakeTimeout]: before test
  1> [2017-04-06T13:59:07,199][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[90]}, bound_addresses {local[90]}
  1> [2017-04-06T13:59:07,200][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[91]}, bound_addresses {local[91]}
  1> [2017-04-06T13:59:07,202][INFO ][o.e.t.l.SimpleLocalTransportTests] [testTcpHandshakeTimeout]: after test
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
IGNOR/A 0.01s J1 | SimpleLocalTransportTests.testTcpHandshakeTimeout
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   > Assumption #1: only tcp transport does a handshake
  1> [2017-04-06T13:59:07,204][INFO ][o.e.t.l.SimpleLocalTransportTests] [testThreadContext]: before test
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> "SUITE-SimpleLocalTransportTests-seed#[290BC1B07436B39B]" ID=857 RUNNABLE
  2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:675)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
  2> 	- locked java.lang.Object@6c514907
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
  2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@655179d5
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@655179d5
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
  2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
  2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@d6b5825
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.Reference$Lock@d6b5825
  2> 	at java.lang.Object.wait(Object.java:502)
  2> 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
  2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
  2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@b93c5a6
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@b93c5a6
  2> 	at java.lang.Thread.join(Thread.java:1249)
  2> 	at java.lang.Thread.join(Thread.java:1323)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
  1> [2017-04-06T13:59:07,207][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[92]}, bound_addresses {local[92]}
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
  1> [2017-04-06T13:59:07,207][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[93]}, bound_addresses {local[93]}
  1> [2017-04-06T13:59:07,215][INFO ][o.e.t.l.SimpleLocalTransportTests] [testThreadContext]: after test
  1> [2017-04-06T13:59:07,217][INFO ][o.e.t.l.SimpleLocalTransportTests] [testConcurrentSendRespondAndDisconnect]: before test
  1> [2017-04-06T13:59:07,219][INFO ][o.e.t.t.MockTransportService] [TS_A] publish_address {local[94]}, bound_addresses {local[94]}
  2> ^^==============================================
  1> [2017-04-06T13:59:07,219][INFO ][o.e.t.t.MockTransportService] [TS_B] publish_address {local[95]}, bound_addresses {local[95]}
  1> [2017-04-06T13:59:07,230][INFO ][o.e.t.t.MockTransportService] [TS_B_0] publish_address {local[96]}, bound_addresses {local[96]}
  1> [2017-04-06T13:59:07,231][INFO ][o.e.t.t.MockTransportService] [TS_B_3] publish_address {local[97]}, bound_addresses {local[97]}
  1> [2017-04-06T13:59:07,231][INFO ][o.e.t.t.MockTransportService] [TS_B_6] publish_address {local[98]}, bound_addresses {local[98]}
  1> [2017-04-06T13:59:07,232][INFO ][o.e.t.t.MockTransportService] [TS_B_9] publish_address {local[99]}, bound_addresses {local[99]}
  2> REPRODUCE WITH: gradle :core:test -Dtests.seed=290BC1B07436B39B -Dtests.class=org.elasticsearch.transport.local.SimpleLocalTransportTests -Dtests.method="testConcurrentSendRespondAndDisconnect" -Dtests.security.manager=true -Dtests.locale=zh-SG -Dtests.timezone=Asia/Aqtau
ERROR   1199s J1 | SimpleLocalTransportTests.testConcurrentSendRespondAndDisconnect <<< FAILURES!
   > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   > 	at __randomizedtesting.SeedInfo.seed([290BC1B07436B39B]:0)
  2> 四月 06, 2017 2:19:06 下午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 11 leaked thread(s).
  2> 四月 06, 2017 2:19:11 下午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 11 threads leaked from SUITE scope at org.elasticsearch.transport.local.SimpleLocalTransportTests: 
  2>    1) Thread[id=858, name=TEST-SimpleLocalTransportTests.testConcurrentSendRespondAndDisconnect-seed#[290BC1B07436B39B], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  1> [2017-04-06T14:19:11,129][ERROR][o.e.t.l.SimpleLocalTransportTests] caught exception while sending to node {TS_B}{TS_B}{GbRouU_HT0GtKrANTCtUyA}{local}{local[95]}
  1> java.lang.IllegalStateException: Future got interrupted
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2>         at org.elasticsearch.transport.AbstractSimpleTransportTestCase.testConcurrentSendRespondAndDisconnect(AbstractSimpleTransportTestCase.java:635)
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  1> 	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:44) ~[main/:?]
  1> 	at org.elasticsearch.transport.AbstractSimpleTransportTestCase$13.doRun(AbstractSimpleTransportTestCase.java:596) ~[framework-5.3.1-SNAPSHOT.jar:?]
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.lang.reflect.Method.invoke(Method.java:498)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
  1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
  1> 	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_121]
  1> Caused by: java.lang.InterruptedException
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_121]
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_121]
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  1> 	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:248) ~[main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:91) ~[main/:?]
  1> 	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42) ~[main/:?]
  1> 	... 6 more
  1> [2017-04-06T14:19:11,131][INFO ][o.e.t.l.SimpleLocalTransportTests] [testConcurrentSendRespondAndDisconnect]: after test
  2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    2) Thread[id=1076, name=elasticsearch[TS_A][local_transport][T#3], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    3) Thread[id=1075, name=elasticsearch[TS_A][local_transport][T#2], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    4) Thread[id=1080, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#12], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    5) Thread[id=1083, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#13], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    6) Thread[id=1070, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#9], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2>         at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:248)
ERROR   0.00s J1 | SimpleLocalTransportTests (suite) <<< FAILURES!
   > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
   > 	at __randomizedtesting.SeedInfo.seed([290BC1B07436B39B]:0)
Completed [868/902] on J1 in 1205.08s, 27 tests, 2 errors, 6 skipped <<< FAILURES!

  2>         at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:91)
  2>         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
  2>         at org.elasticsearch.transport.AbstractSimpleTransportTestCase$13.doRun(AbstractSimpleTransportTestCase.java:596)
  2>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
  2>         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    7) Thread[id=1084, name=elasticsearch[TS_B_9][local_transport][T#1], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    8) Thread[id=1079, name=elasticsearch[TS_A][local_transport][T#4], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>    9) Thread[id=1061, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][[timer]], state=TIMED_WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at java.lang.Thread.sleep(Native Method)
  2>         at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)
  2>   10) Thread[id=1065, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#4], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
Suite: org.elasticsearch.cluster.routing.allocation.AllocateUnassignedDecisionTests
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2>   11) Thread[id=1073, name=elasticsearch[TS_A][local_transport][T#1], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>         at sun.misc.Unsafe.park(Native Method)
Completed [869/902] on J1 in 0.03s, 6 tests

  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2>         at java.lang.Thread.run(Thread.java:745)
  2> 四月 06, 2017 2:19:11 下午 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=858, name=TEST-SimpleLocalTransportTests.testConcurrentSendRespondAndDisconnect-seed#[290BC1B07436B39B], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    2) Thread[id=1076, name=elasticsearch[TS_A][local_transport][T#3], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    3) Thread[id=1075, name=elasticsearch[TS_A][local_transport][T#2], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    4) Thread[id=1080, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#12], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    5) Thread[id=1083, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#13], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    6) Thread[id=1070, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#9], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    7) Thread[id=1084, name=elasticsearch[TS_B_9][local_transport][T#1], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    8) Thread[id=1079, name=elasticsearch[TS_A][local_transport][T#4], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>    9) Thread[id=1061, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][[timer]], state=TIMED_WAITING, group=TGRP-SimpleLocalTransportTests]
  2>   10) Thread[id=1065, name=elasticsearch[org.elasticsearch.transport.local.SimpleLocalTransportTests][generic][T#4], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2>   11) Thread[id=1073, name=elasticsearch[TS_A][local_transport][T#1], state=WAITING, group=TGRP-SimpleLocalTransportTests]
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+5.3+multijob-unix-compatibility/os/fedora/core/build/testrun/test/J1/temp/org.elasticsearch.transport.local.SimpleLocalTransportTests_290BC1B07436B39B-001
  2> Apr 06, 2017 9:19:11 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
  2> REPRODUCE WITH: gradle :core:test -Dtests.seed=290BC1B07436B39B -Dtests.class=org.elasticsearch.transport.local.SimpleLocalTransportTests -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC
  2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=zh-SG, timezone=Asia/Aqtau
  2> NOTE: Linux 4.9.14-200.fc25.x86_64 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=395954336,total=531628032
  2> NOTE: All tests run in this JVM: [WriteableIngestDocumentTests, MatchPhrasePrefixQueryBuilderTests, HasParentQueryBuilderTests, OperationRoutingTests,
@martijnvg martijnvg added :Distributed/Network Http and internode communication implementations >test Issues or PRs that are addressing/adding tests labels Apr 6, 2017
@bleskes bleskes assigned s1monw and unassigned bleskes Apr 6, 2017
s1monw added a commit to s1monw/elasticsearch that referenced this issue Apr 7, 2017
@s1monw
Fix possible hang in local transport when nodes get concurrently disc…
ae1158c 
…onnected

There is a very small race that can cause requests to hang if nodes are concurrently
disconnected while a connection is fetched form the transport and the client handler
is installed in the TransportService. The code relies on the fact that closed connections
either fully work or fail and cause the requests to be ended / finished with an exception.
This is not necessarily true in local transport since it doesn't really maintain connections
in the classical sense.
This change adds additional checking for disconnected nodes when the connection is used to ensure
the conneciton is never used with a disconnected transport.

Closes elastic#23942
@s1monw s1monw mentioned this issue Apr 7, 2017
Merged
s1monw added a commit that referenced this issue Apr 7, 2017
@s1monw
Fix possible hang in local transport when nodes get concurrently disc…
2c65efb 
…onnected (#23962)

There is a very small race that can cause requests to hang if nodes are concurrently
disconnected while a connection is fetched form the transport and the client handler
is installed in the TransportService. The code relies on the fact that closed connections
either fully work or fail and cause the requests to be ended / finished with an exception.
This is not necessarily true in local transport since it doesn't really maintain connections
in the classical sense.
This change adds additional checking for disconnected nodes when the connection is used to ensure
the connection is never used with a disconnected transport.

Closes #23942
s1monw added a commit that referenced this issue Apr 7, 2017
@s1monw
Fix possible hang in local transport when nodes get concurrently disc…
500a9e2 
…onnected (#23962)

There is a very small race that can cause requests to hang if nodes are concurrently
disconnected while a connection is fetched form the transport and the client handler
is installed in the TransportService. The code relies on the fact that closed connections
either fully work or fail and cause the requests to be ended / finished with an exception.
This is not necessarily true in local transport since it doesn't really maintain connections
in the classical sense.
This change adds additional checking for disconnected nodes when the connection is used to ensure
the connection is never used with a disconnected transport.

Closes #23942
s1monw added a commit that referenced this issue Apr 7, 2017
@s1monw
Fix possible hang in local transport when nodes get concurrently disc…
4185901 
…onnected (#23962)

There is a very small race that can cause requests to hang if nodes are concurrently
disconnected while a connection is fetched form the transport and the client handler
is installed in the TransportService. The code relies on the fact that closed connections
either fully work or fail and cause the requests to be ended / finished with an exception.
This is not necessarily true in local transport since it doesn't really maintain connections
in the classical sense.
This change adds additional checking for disconnected nodes when the connection is used to ensure
the connection is never used with a disconnected transport.

Closes #23942
@s1monw
Copy link
Contributor

s1monw commented May 19, 2017

Fixed by ae1158c

@s1monw s1monw closed this as completed May 19, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s1monw s1monw

Labels
:Distributed/Network Http and internode communication implementations >test Issues or PRs that are addressing/adding tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@martijnvg @s1monw @bleskes