TestRaptorDistributedQueriesBucketed::init is flaky

[flakycov]

TestRaptorDistributedQueriesBucketed::init failed for us on commit 56ea02d but did not fail in the corresponding run on Travis:

init(com.facebook.presto.raptor.TestRaptorDistributedQueriesBucketed)  Time elapsed: 9.224 sec  <<< FAILURE!
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error notifying InjectionListener io.airlift.bootstrap.LifeCycleModule$1$$Lambda$150/368676592@2e71e1ab of io.airlift.http.server.testing.TestingHttpServer.
 Reason: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
  at io.airlift.http.server.testing.TestingHttpServerModule.configure(TestingHttpServerModule.java:58)
  while locating io.airlift.http.server.testing.TestingHttpServer
  while locating io.airlift.http.server.HttpServer
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at io.airlift.bootstrap.LifeCycleModule$1.lambda$hear$0(LifeCycleModule.java:64)
	at com.google.inject.internal.MembersInjectorImpl.notifyListeners(MembersInjectorImpl.java:119)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:115)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
	at com.google.inject.Guice.createInjector(Guice.java:96)
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:242)
	at com.facebook.presto.server.testing.TestingPrestoServer.<init>(TestingPrestoServer.java:238)
	at com.facebook.presto.tests.DistributedQueryRunner.createTestingPrestoServer(DistributedQueryRunner.java:183)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:122)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:91)
	at com.facebook.presto.raptor.RaptorQueryRunner.createRaptorQueryRunner(RaptorQueryRunner.java:49)
	at com.facebook.presto.raptor.TestRaptorDistributedQueriesBucketed.lambda$new$0(TestRaptorDistributedQueriesBucketed.java:26)
	at com.facebook.presto.tests.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.airlift.bootstrap.LifeCycleManager.startInstance(LifeCycleManager.java:171)
	at io.airlift.bootstrap.LifeCycleManager.addInstance(LifeCycleManager.java:157)
	at io.airlift.bootstrap.LifeCycleModule$1.lambda$hear$0(LifeCycleModule.java:61)
	... 38 more
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.server.Server.doStart(Server.java:390)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at io.airlift.http.server.HttpServer.start(HttpServer.java:381)
	... 45 more

2) Error notifying InjectionListener io.airlift.bootstrap.LifeCycleModule$1$$Lambda$150/368676592@2e71e1ab of io.airlift.http.server.testing.TestingHttpServer.
 Reason: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
  at io.airlift.http.server.testing.TestingHttpServerModule.configure(TestingHttpServerModule.java:58)
  while locating io.airlift.http.server.testing.TestingHttpServer
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at io.airlift.bootstrap.LifeCycleModule$1.lambda$hear$0(LifeCycleModule.java:64)
	at com.google.inject.internal.MembersInjectorImpl.notifyListeners(MembersInjectorImpl.java:119)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:115)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
	at com.google.inject.Guice.createInjector(Guice.java:96)
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:242)
	at com.facebook.presto.server.testing.TestingPrestoServer.<init>(TestingPrestoServer.java:238)
	at com.facebook.presto.tests.DistributedQueryRunner.createTestingPrestoServer(DistributedQueryRunner.java:183)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:122)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:91)
	at com.facebook.presto.raptor.RaptorQueryRunner.createRaptorQueryRunner(RaptorQueryRunner.java:49)
	at com.facebook.presto.raptor.TestRaptorDistributedQueriesBucketed.lambda$new$0(TestRaptorDistributedQueriesBucketed.java:26)
	at com.facebook.presto.tests.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.airlift.bootstrap.LifeCycleManager.startInstance(LifeCycleManager.java:171)
	at io.airlift.bootstrap.LifeCycleManager.addInstance(LifeCycleManager.java:157)
	at io.airlift.bootstrap.LifeCycleModule$1.lambda$hear$0(LifeCycleModule.java:61)
	... 37 more
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.server.Server.doStart(Server.java:390)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at io.airlift.http.server.HttpServer.start(HttpServer.java:381)
	... 44 more

2 errors
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
	at com.google.inject.Guice.createInjector(Guice.java:96)
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:242)
	at com.facebook.presto.server.testing.TestingPrestoServer.<init>(TestingPrestoServer.java:238)
	at com.facebook.presto.tests.DistributedQueryRunner.createTestingPrestoServer(DistributedQueryRunner.java:183)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:122)
	at com.facebook.presto.tests.DistributedQueryRunner.<init>(DistributedQueryRunner.java:91)
	at com.facebook.presto.raptor.RaptorQueryRunner.createRaptorQueryRunner(RaptorQueryRunner.java:49)
	at com.facebook.presto.raptor.TestRaptorDistributedQueriesBucketed.lambda$new$0(TestRaptorDistributedQueriesBucketed.java:26) 

Our investigation shows that this test is likely flaky. We set up our build environment to be as close to your Travis environment as possible, using Ubuntu 14.04.5 LTS, Java HotSpot 1.8.0_111-b14 and Maven 3.3.9.

[ArturGajowy]

@flakycov what are you? Some kind of a bot/service? Tool? I'm intrigued and would like to know more. I googled but found no info.

[flakycov]

We're not a bot :)

We're a group of (academic) software engineering researchers studying flaky tests in the wild - how to detect them and how to debug them. We hope that these reports are useful (we had started with just one report, and then today added the rest after the comment from electrum yesterday.

We'd be happy to discuss more, including our mechanism for finding these flaky tests (and share who we are) with you or any other developers who are interested.

Since we are currently preparing this work to be submitted to a conference that follows double-blind review (i.e. author names are anonymized), we are trying to be careful not to leave many public clues across the internet (like in an issue tracker) that point to who we are at this point. However, feel free to send us an email - flakycov@gmail.com and we'll respond from our personal accounts with details.

[electrum]

All the "Address already in use" errors will be fixed by airlift/airlift#539