Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Timeouts in SSLConfigurationReloaderTests on 5.6 #36053

Closed
cbuescher opened this issue Nov 29, 2018 · 3 comments
Closed

[CI] Timeouts in SSLConfigurationReloaderTests on 5.6 #36053

cbuescher opened this issue Nov 29, 2018 · 3 comments
Assignees
@jaymode
Labels
:Security/TLS SSL/TLS, Certificates >test-failure Triaged test failures from CI v5.6.14

Comments

@cbuescher
Copy link
Member

https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+5.6+multijob-unix-compatibility/469/console

2> WARNING: Suite execution timed out: org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests
  2> ==== jstack at approximately timeout time ====
  1> [2018-11-28T19:15:59,837][INFO ][o.e.x.s.SSLConfigurationReloaderTests] [testReloadingTrustStore]: before test
  1> [2018-11-28T19:16:00,839][INFO ][o.e.x.s.SSLConfigurationReloaderTests$5] reloaded [/var/lib/jenkins/workspace/elastic+x-pack-elasticsearch+5.6+multijob-unix-compatibility/elasticsearch-extra/x-pack-elasticsearch/plugin/build/testrun/test/J2/temp/org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests_1ED4047765A4F425-001/tempDir-004/testnode.jks] and updated ssl contexts using this file
  2> "elasticsearch[reload tests][scheduler][T#1]" ID=8712 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@505d9971
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@505d9971
  1> [2018-11-28T19:16:00,840][INFO ][o.e.x.s.SSLConfigurationReloaderTests] [testReloadingTrustStore]: after test
  2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[reload tests][[timer]]" ID=8711 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)
  2> "TEST-SSLConfigurationReloaderTests.testPEMKeyConfigReloading-seed#[1ED4047765A4F425]" ID=8706 WAITING on java.util.concurrent.CountDownLatch$Sync@74a8ccba
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@74a8ccba
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2> 	at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.validateSSLConfigurationIsReloaded(SSLConfigurationReloaderTests.java:534)
  2> 	at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.validateKeyConfigurationIsReloaded(SSLConfigurationReloaderTests.java:469)
  2> 	at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.testPEMKeyConfigReloading(SSLConfigurationReloaderTests.java:209)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  1> [2018-11-28T19:16:00,842][INFO ][o.e.x.s.SSLConfigurationReloaderTests] [testPEMKeyConfigReloading]: before test
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "SUITE-SSLConfigurationReloaderTests-seed#[1ED4047765A4F425]" ID=8705 RUNNABLE
  2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
  2> 	- locked java.lang.Object@6147eba6
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
  2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@74f0a6a5
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@74f0a6a5
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
  2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
  2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@6120de4b
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.Reference$Lock@6120de4b
  2> 	at java.lang.Object.wait(Object.java:502)
  2> 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
  2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
  2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1160934a
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1160934a
  2> 	at java.lang.Thread.join(Thread.java:1252)
  2> 	at java.lang.Thread.join(Thread.java:1326)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
  2> ^^==============================================
  2> REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:test -Dtests.seed=1ED4047765A4F425 -Dtests.class=org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests -Dtests.method="testPEMKeyConfigReloading" -Dtests.security.manager=true -Dtests.locale=ar-QA -Dtests.timezone=America/Kralendijk
ERROR   1199s J2 | SSLConfigurationReloaderTests.testPEMKeyConfigReloading <<< FAILURES!
  2> نوف 28, 2018 7:35:59 م com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
  2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
  2> نوف 28, 2018 7:36:04 م com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 3 threads leaked from SUITE scope at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests: 
  2>    1) Thread[id=8706, name=TEST-SSLConfigurationReloaderTests.testPEMKeyConfigReloading-seed#[1ED4047765A4F425], state=WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2>         at sun.misc.Unsafe.park(Native Method)
   > 	at __randomizedtesting.SeedInfo.seed([1ED4047765A4F425]:0)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2>         at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.validateSSLConfigurationIsReloaded(SSLConfigurationReloaderTests.java:534)
  2>         at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.validateKeyConfigurationIsReloaded(SSLConfigurationReloaderTests.java:469)
  2>         at org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests.testPEMKeyConfigReloading(SSLConfigurationReloaderTests.java:209)
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.lang.reflect.Method.invoke(Method.java:498)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  1> [2018-11-28T19:36:04,903][INFO ][o.e.x.s.SSLConfigurationReloaderTests] [testPEMKeyConfigReloading]: after test
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    2) Thread[id=8712, name=elasticsearch[reload tests][scheduler][T#1], state=TIMED_WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    3) Thread[id=8711, name=elasticsearch[reload tests][[timer]], state=TIMED_WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2>         at java.lang.Thread.sleep(Native Method)
ERROR   0.00s J2 | SSLConfigurationReloaderTests (suite) <<< FAILURES!
   > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
   > 	at __randomizedtesting.SeedInfo.seed([1ED4047765A4F425]:0)
Completed [645/645] on J2 in 1205.09s, 3 tests, 2 errors <<< FAILURES!

  2>         at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)
  2> نوف 28, 2018 7:36:04 م com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=8706, name=TEST-SSLConfigurationReloaderTests.testPEMKeyConfigReloading-seed#[1ED4047765A4F425], state=WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2>    2) Thread[id=8712, name=elasticsearch[reload tests][scheduler][T#1], state=TIMED_WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2>    3) Thread[id=8711, name=elasticsearch[reload tests][[timer]], state=TIMED_WAITING, group=TGRP-SSLConfigurationReloaderTests]
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+x-pack-elasticsearch+5.6+multijob-unix-compatibility/elasticsearch-extra/x-pack-elasticsearch/plugin/build/testrun/test/J2/temp/org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests_1ED4047765A4F425-001
  2> Nov 28, 2018 11:36:04 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.

I couldn't reproduce locally so far.

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:test -Dtests.seed=1ED4047765A4F425 -Dtests.class=org.elasticsearch.xpack.ssl.SSLConfigurationReloaderTests -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC
@cbuescher cbuescher added >test-failure Triaged test failures from CI :Security/TLS SSL/TLS, Certificates v5.6.14 labels Nov 29, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@danielmitterdorfer
Copy link
Member

We looked at this one in test triage power hour and we noticed two things:

  1. We should wrap the test code that counts down the latch in a ' try ... finally` block because it might be the case that an exception occurs while reloading and then the latch would never count down exhibiting the behaviour that we see in the test failure.
  2. It's possible that an exception occurs in the following code but it is only logged on trace level so we should raise the log level for that test.

elasticsearch/core/src/main/java/org/elasticsearch/watcher/ResourceWatcherService.java

Line 199 in d7c419f

logger.trace("failed to check resource watcher", e);

@jaymode jaymode self-assigned this Jan 10, 2019
jaymode added a commit to jaymode/elasticsearch that referenced this issue Jan 10, 2019
@jaymode
Ensure latch is counted down in ssl reload test
3943d9b 
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See elastic#36053
@jaymode jaymode mentioned this issue Jan 10, 2019
Merged
jaymode added a commit that referenced this issue Jan 10, 2019
@jaymode
Ensure latch is counted down in ssl reload test (#37313)
e6d3d85 
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See #36053
jaymode added a commit that referenced this issue Jan 10, 2019
@jaymode
Ensure latch is counted down in ssl reload test (#37313)
f1d3ee8 
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See #36053
jaymode added a commit that referenced this issue Jan 10, 2019
@jaymode
Ensure latch is counted down in ssl reload test (#37313)
00b105c 
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See #36053
jaymode added a commit that referenced this issue Jan 10, 2019
@jaymode
Ensure latch is counted down in ssl reload test (#37313)
c7316e0 
This change ensures we always countdown the latch in the
SSLConfigurationReloaderTests to prevent the suite from timing out in
case of an exception. Additionally, we also increase the logging of the
resource watcher in case an IOException occurs.

See #36053
@jkakavas
Copy link
Member

Closing this as it should be resolved by #37313

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaymode jaymode

Labels
:Security/TLS SSL/TLS, Certificates >test-failure Triaged test failures from CI v5.6.14
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@danielmitterdorfer @jaymode @jkakavas @cbuescher @elasticmachine