Skip to content

/ arquillian-cube

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.io.IOException: Failed to upgrade connection when running test against OCP 3.9 #1048

Closed
goldmann opened this issue Apr 19, 2018 · 2 comments
Closed

java.io.IOException: Failed to upgrade connection when running test against OCP 3.9 #1048

goldmann opened this issue Apr 19, 2018 · 2 comments
Milestone
1.16.0

Comments

@goldmann
Copy link
Contributor

@goldmann goldmann commented Apr 19, 2018

Issue Overview

I would like to get some help on investigating why this is failing on newest OCP release. This can be reproduced every time. I'm using the 1.15.4-SNAPSHOT version which is current master + #1046. It looks like the test is executed successfully, environment teared down correctly. I do remember tests passing fine on this environment, but now these are failing. I'm not entirely sure why it is this case.

Any ideas?

Additional Information

I do see also this in testrunner log:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get https://172.16.212.13:10250/containerLogs/test-sso-7f61f278/testrunner/testrunner?follow=true: dial tcp 172.16.212.13:10250: getsockopt: connection timed out","code":500}

And in the output log:

INFO: Deleting environment for org.jboss.test.arquillian.ce.sso.Sso72ServerBasicTest
Exception in thread "OkHttp Dispatcher" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1282bda0 rejected from java.util.concurrent.ThreadPoolExecutor@637dd22a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
	at io.fabric8.kubernetes.clnt.v3_1.dsl.internal.LogWatchCallback.onResponse(LogWatchCallback.java:162)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-04-19T08:26:30Z] [Normal] [Pod:testrunner]: (Killing) Killing container with id docker://testrunner:Need to kill Pod

[2018-04-19T08:26:30Z] [Normal] [Pod:testrunner]: (Killing) Killing container with id docker://testrunner:Need to kill Pod

This is running the test on:

OpenShift Master: v3.9.14
Kubernetes Master: v1.9.1+a0ce1bc657
OpenShift Web Console: v3.9.14

Stacktrace can be found below.

Stacktrace 
Running org.jboss.test.arquillian.ce.sso.Sso72ServerLogTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 159.383 sec <<< FAILURE! - in org.jboss.test.arquillian.ce.sso.Sso72ServerLogTest
org.jboss.test.arquillian.ce.sso.Sso72ServerLogTest  Time elapsed: 159.383 sec  <<< ERROR!
org.arquillian.cube.spi.CubeControlException: Could not start testrunner
	at org.arquillian.cube.spi.CubeControlException.failedStart(CubeControlException.java:19)
	at org.arquillian.cube.openshift.impl.model.BuildablePodCube.start(BuildablePodCube.java:128)
	at org.arquillian.cube.impl.client.CubeLifecycleController.start(CubeLifecycleController.java:19)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
	at org.arquillian.cube.impl.client.container.CubeContainerLifecycleController.startCubeMappedContainer(CubeContainerLifecycleController.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
	at org.jboss.arquillian.container.impl.ContainerImpl.start(ContainerImpl.java:177)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:137)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:133)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:208)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startContainer(ContainerLifecycleController.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$3.perform(ContainerLifecycleController.java:82)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$3.perform(ContainerLifecycleController.java:76)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forEachClassContainer(ContainerLifecycleController.java:191)
	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startClassContainers(ContainerLifecycleController.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:62)
	at org.jboss.arquillian.container.test.impl.client.ContainerEventController.execute(ContainerEventController.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)
	at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:83)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105)
	at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.beforeClass(EventTestRunnerAdaptor.java:89)
	at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:163)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:350)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:54)
	at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:177)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:115)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: java.io.IOException: Failed to establish portforward client connection
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder.(PortForwarder.java:145)
	at org.arquillian.cube.openshift.impl.model.BuildablePodCube$PortBindings.createProxies(BuildablePodCube.java:357)
	at org.arquillian.cube.openshift.impl.model.BuildablePodCube$PortBindings.podStarted(BuildablePodCube.java:349)
	at org.arquillian.cube.openshift.impl.model.BuildablePodCube$PortBindings.access$000(BuildablePodCube.java:214)
	at org.arquillian.cube.openshift.impl.model.BuildablePodCube.start(BuildablePodCube.java:114)
	... 96 more
Caused by: java.io.IOException: Unexpected error
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder$2$1.completed(PortForwarder.java:123)
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder$2$1.completed(PortForwarder.java:117)
	at io.undertow.client.http.HttpClientExchange.setResponse(HttpClientExchange.java:115)
	at io.undertow.client.http.HttpClientConnection$ClientReadListener.handleEvent(HttpClientConnection.java:584)
	at io.undertow.client.http.HttpClientConnection$ClientReadListener.handleEvent(HttpClientConnection.java:473)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
Caused by: java.io.IOException: Failed to upgrade connection
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder.upgradeConnection(PortForwarder.java:262)
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder.access$000(PortForwarder.java:48)
	at org.arquillian.cube.kubernetes.impl.portforward.PortForwarder$2$1.completed(PortForwarder.java:121)
	... 8 more
@goldmann
Copy link
Contributor Author

@goldmann goldmann commented Apr 19, 2018

Issue may be related to some OCP hiccup, I cannot get even logs using web console or oc utility. Investigating.
@goldmann
Copy link
Contributor Author

@goldmann goldmann commented Apr 19, 2018

Closing this issue. The problem was that in my environment, I accidentally disabled required ports for logs fetching and this caused the mentioned issue.
@goldmann goldmann closed this Apr 19, 2018
@dipak-pawar dipak-pawar added this to the 1.16.0 milestone Jun 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.16.0
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@goldmann @dipak-pawar