Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] HttpClientTests.testThatGetRedirectIsFollowed failure #36546

Closed
astefan opened this issue Dec 12, 2018 · 3 comments
Closed

[CI] HttpClientTests.testThatGetRedirectIsFollowed failure #36546

astefan opened this issue Dec 12, 2018 · 3 comments
Assignees
@dakrone
Labels
:Data Management/Watcher >test-failure Triaged test failures from CI

Comments

@astefan
Copy link
Contributor

astefan commented Dec 12, 2018

Didn't manage to reproduce this one locally: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=ubuntu&&virtual/113/console

gradlew :x-pack:plugin:watcher:test -Dtests.seed=6D5B520AF528F071 -Dtests.class=org.elasticsearch.xpack.watcher.common.http.HttpClientTests -Dtests.method="testThatGetRedirectIsFollowed" -Dtests.security.manager=true -Dtests.locale=lv-LV -Dtests.timezone=America/Campo_Grande -Dcompiler.java=11 -Druntime.java=8

java.net.SocketTimeoutException: Read timed out
	at __randomizedtesting.SeedInfo.seed([6D5B520AF528F071:1217D9506799D023]:0)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.elasticsearch.xpack.watcher.common.http.HttpClient.lambda$execute$0(HttpClient.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:32)
	at org.elasticsearch.xpack.watcher.common.http.HttpClient.execute(HttpClient.java:172)
	at org.elasticsearch.xpack.watcher.common.http.HttpClientTests.testThatGetRedirectIsFollowed(HttpClientTests.java:571)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
@astefan astefan added >test-failure Triaged test failures from CI :Data Management/Watcher labels Dec 12, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@spinscale
Copy link
Contributor

relevant snippet from the logs. I do not have an idea yet, how the mock web server can bind to a port, but apparently is not reachable within 10 seconds.

  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:watcher:test -Dtests.seed=6D5B520AF528F071 -Dtests.class=org.elasticsearch.xpack.watcher.common.http.HttpClientTests -Dtests.method="testThatGetRedirectIsFollowed" -Dtests.security.manager=true -Dtests.local
e=lv-LV -Dtests.timezone=America/Campo_Grande -Dcompiler.java=11 -Druntime.java=8


  1> [2018-12-12T05:01:20,156][INFO ][o.e.x.w.c.h.HttpClientTests] [testThatGetRedirectIsFollowed] before test
  1> [2018-12-12T05:01:20,157][INFO ][o.e.t.h.MockWebServer    ] [testThatGetRedirectIsFollowed] bound HTTP mock server to [127.0.0.1:36632]
  1> [2018-12-12T05:01:30,171][INFO ][o.e.x.w.c.h.HttpClientTests] [testThatGetRedirectIsFollowed] after test
ERROR   10.0s J1 | HttpClientTests.testThatGetRedirectIsFollowed <<< FAILURES!
   > Throwable #1: java.net.SocketTimeoutException: Read timed out
   >    at __randomizedtesting.SeedInfo.seed([6D5B520AF528F071:1217D9506799D023]:0)
   >    at java.net.SocketInputStream.socketRead0(Native Method)
   >    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   >    at java.net.SocketInputStream.read(SocketInputStream.java:171)
   >    at java.net.SocketInputStream.read(SocketInputStream.java:141)
   >    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   >    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   >    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   >    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   >    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   >    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   >    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   >    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
   >    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   >    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   >    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
   >    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   >    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   >    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   >    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   >    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   >    at org.elasticsearch.xpack.watcher.common.http.HttpClient.lambda$execute$0(HttpClient.java:172)
   >    at java.security.AccessController.doPrivileged(Native Method)
   >    at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:32)
   >    at org.elasticsearch.xpack.watcher.common.http.HttpClient.execute(HttpClient.java:172)
   >    at org.elasticsearch.xpack.watcher.common.http.HttpClientTests.testThatGetRedirectIsFollowed(HttpClientTests.java:571)
   >    at java.lang.Thread.run(Thread.java:748)
IGNOR/A 0.00s J1 | HttpClientTests.testHttpsWithoutTruststore

@dakrone dakrone self-assigned this Feb 1, 2019
@dakrone
Copy link
Member

dakrone commented Feb 1, 2019

I ran these tests over 10,000 times without reproducing this timeout, at this point I don't think we can do anything about this since it would require wrapping all the calls in a retry (which would be very ugly). I'm going to close this for now and if it happens again we can revisit.

@dakrone dakrone closed this as completed Feb 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakrone dakrone

Labels
:Data Management/Watcher >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dakrone @spinscale @astefan @elasticmachine