Skip to content

[QoB] new transient error: java.net.SocketTimeoutException: connect timed out #13074

New issue
New issue
Closed
#13075
Closed
[QoB] new transient error: java.net.SocketTimeoutException: connect timed out#13074
#13075
Assignees
danking
Labels
bug
@danking

Description

@danking
danking
opened on May 17, 2023

What happened?

https://ci.hail.is/batches/7450431/jobs/109

After #13043 merges, we'll always recur through the causal chain so we just need to retry java.net.SocketTimeoutException: connect timed out.

E           is.hail.relocated.org.apache.http.conn.ConnectTimeoutException: Connect to internal.hail:80 [internal.hail/10.128.0.57] failed: connect timed out
E           	at is.hail.relocated.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
E           	at is.hail.relocated.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
E           	at is.hail.relocated.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
E           	at is.hail.relocated.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
E           	at is.hail.relocated.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
E           	at is.hail.relocated.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
E           	at is.hail.relocated.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
E           	at is.hail.relocated.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
E           	at is.hail.relocated.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
E           	at is.hail.relocated.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
E           	at is.hail.services.Requester.$anonfun$requestWithHandler$1(Requester.scala:79)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.services.Requester.requestWithHandler(Requester.scala:78)
E           	at is.hail.services.Requester.request(Requester.scala:104)
E           	at is.hail.services.batch_client.BatchClient.post(BatchClient.scala:60)
E           	at is.hail.services.batch_client.BatchClient.$anonfun$update$1(BatchClient.scala:92)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.services.batch_client.BatchClient.update(BatchClient.scala:90)
E           	at is.hail.backend.service.ServiceBackend.parallelizeAndComputeWithIndex(ServiceBackend.scala:205)
E           	at is.hail.backend.BackendUtils.collectDArray(BackendUtils.scala:45)
E           	at __C229012Compiled.__m229048split_CollectDistributedArray(Emit.scala)
E           	at __C229012Compiled.__m229014split_Let(Emit.scala)
E           	at __C229012Compiled.apply(Emit.scala)
E           	at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$7(CompileAndEvaluate.scala:74)
E           	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:74)
E           	at is.hail.expr.ir.CompileAndEvaluate$.evalToIR(CompileAndEvaluate.scala:33)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.execute$1(EvalRelationalLets.scala:12)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.lower$1(EvalRelationalLets.scala:18)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.apply(EvalRelationalLets.scala:37)
E           	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.transform(LoweringPass.scala:147)
E           	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14)
E           	at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13)
E           	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.apply(LoweringPass.scala:141)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:22)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:20)
E           	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
E           	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
E           	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:20)
E           	at is.hail.backend.service.ServiceBackend.execute(ServiceBackend.scala:313)
E           	at is.hail.backend.service.ServiceBackend.execute(ServiceBackend.scala:349)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$12(ServiceBackend.scala:702)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.withIRFunctionsReadFromInput(ServiceBackend.scala:805)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$11(ServiceBackend.scala:700)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$2(ServiceBackend.scala:658)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:75)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:75)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17)
E           	at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:63)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$1(ServiceBackend.scala:648)
E           	at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52)
E           	at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.withExecuteContext$1(ServiceBackend.scala:634)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.executeOneCommand(ServiceBackend.scala:697)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$6(ServiceBackend.scala:462)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$6$adapted(ServiceBackend.scala:461)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$5(ServiceBackend.scala:461)
E           	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$4(ServiceBackend.scala:460)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$4$adapted(ServiceBackend.scala:459)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$3(ServiceBackend.scala:459)
E           	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.main(ServiceBackend.scala:458)
E           	at is.hail.backend.service.Main$.main(Main.scala:15)
E           	at is.hail.backend.service.Main.main(Main.scala)
E           	at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
E           	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E           	at java.lang.reflect.Method.invoke(Method.java:498)
E           	at is.hail.JVMEntryway$1.run(JVMEntryway.java:119)
E           	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E           	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E           	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E           	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E           	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E           	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E           	at java.lang.Thread.run(Thread.java:750)
E           
E           java.net.SocketTimeoutException: connect timed out
E           	at java.net.PlainSocketImpl.socketConnect(Native Method)
E           	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
E           	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
E           	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
E           	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
E           	at java.net.Socket.connect(Socket.java:607)
E           	at is.hail.relocated.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
E           	at is.hail.relocated.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
E           	at is.hail.relocated.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
E           	at is.hail.relocated.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
E           	at is.hail.relocated.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
E           	at is.hail.relocated.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
E           	at is.hail.relocated.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
E           	at is.hail.relocated.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
E           	at is.hail.relocated.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
E           	at is.hail.relocated.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
E           	at is.hail.relocated.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
E           	at is.hail.services.Requester.$anonfun$requestWithHandler$1(Requester.scala:79)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.services.Requester.requestWithHandler(Requester.scala:78)
E           	at is.hail.services.Requester.request(Requester.scala:104)
E           	at is.hail.services.batch_client.BatchClient.post(BatchClient.scala:60)
E           	at is.hail.services.batch_client.BatchClient.$anonfun$update$1(BatchClient.scala:92)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.services.batch_client.BatchClient.update(BatchClient.scala:90)
E           	at is.hail.backend.service.ServiceBackend.parallelizeAndComputeWithIndex(ServiceBackend.scala:205)
E           	at is.hail.backend.BackendUtils.collectDArray(BackendUtils.scala:45)
E           	at __C229012Compiled.__m229048split_CollectDistributedArray(Emit.scala)
E           	at __C229012Compiled.__m229014split_Let(Emit.scala)
E           	at __C229012Compiled.apply(Emit.scala)
E           	at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$7(CompileAndEvaluate.scala:74)
E           	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:74)
E           	at is.hail.expr.ir.CompileAndEvaluate$.evalToIR(CompileAndEvaluate.scala:33)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.execute$1(EvalRelationalLets.scala:12)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.lower$1(EvalRelationalLets.scala:18)
E           	at is.hail.expr.ir.lowering.EvalRelationalLets$.apply(EvalRelationalLets.scala:37)
E           	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.transform(LoweringPass.scala:147)
E           	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16)
E           	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)
E           	at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14)
E           	at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13)
E           	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.apply(LoweringPass.scala:141)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:22)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:20)
E           	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
E           	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
E           	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
E           	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:20)
E           	at is.hail.backend.service.ServiceBackend.execute(ServiceBackend.scala:313)
E           	at is.hail.backend.service.ServiceBackend.execute(ServiceBackend.scala:349)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$12(ServiceBackend.scala:702)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.withIRFunctionsReadFromInput(ServiceBackend.scala:805)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$11(ServiceBackend.scala:700)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$2(ServiceBackend.scala:658)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:75)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:75)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17)
E           	at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:63)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.$anonfun$executeOneCommand$1(ServiceBackend.scala:648)
E           	at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52)
E           	at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.withExecuteContext$1(ServiceBackend.scala:634)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2.executeOneCommand(ServiceBackend.scala:697)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$6(ServiceBackend.scala:462)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$6$adapted(ServiceBackend.scala:461)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$5(ServiceBackend.scala:461)
E           	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$4(ServiceBackend.scala:460)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$4$adapted(ServiceBackend.scala:459)
E           	at is.hail.utils.package$.using(package.scala:635)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.$anonfun$main$3(ServiceBackend.scala:459)
E           	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
E           	at is.hail.services.package$.retryTransientErrors(package.scala:134)
E           	at is.hail.backend.service.ServiceBackendSocketAPI2$.main(ServiceBackend.scala:458)
E           	at is.hail.backend.service.Main$.main(Main.scala:15)
E           	at is.hail.backend.service.Main.main(Main.scala)
E           	at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
E           	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E           	at java.lang.reflect.Method.invoke(Method.java:498)
E           	at is.hail.JVMEntryway$1.run(JVMEntryway.java:119)
E           	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E           	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E           	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E           	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E           	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E           	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E           	at java.lang.Thread.run(Thread.java:750)

Version

0.2.116

Relevant log output

No response

Metadata

Metadata

Assignees

Labels

bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions