JES error causes cromwell to hang and spew useless (for a user) error messages #1631

LeeTL1220 · 2016-10-27T13:13:13Z

In order to fix this, I had to issue a few kill -9 ...

.........<snip>.....
[2016-10-27 13:10:07,41] [info] MaterializeWorkflowDescriptorActor [6f995b2d]: Call-to-Backend assignments: case_gatk_acnv_workflow.TumorCalculateTargetCoverage -> JES, case_gatk_acnv_workflow.NormalCalculateTargetCoverage -> JES, case_gatk_acnv_workflow.TumorAnnotateTargets -> JES, case_gatk_acnv_workflow.TumorWholeGenomeCoverage -> JES, case_gatk_acnv_workflow.NormalCaller -> JES, case_gatk_acnv_workflow.PlotACNVResults -> JES, case_gatk_acnv_workflow.TumorNormalizeSomaticReadCounts -> JES, case_gatk_acnv_workflow.NormalWholeGenomeCoverage -> JES, case_gatk_acnv_workflow.NormalAnnotateTargets -> JES, case_gatk_acnv_workflow.CNLoHAndSplitsCaller -> JES, case_gatk_acnv_workflow.TumorPerformSeg -> JES, case_gatk_acnv_workflow.TumorCaller -> JES, case_gatk_acnv_workflow.AllelicCNV -> JES, case_gatk_acnv_workflow.NormalCorrectGCBias -> JES, case_gatk_acnv_workflow.NormalPerformSeg -> JES, case_gatk_acnv_workflow.NormalNormalizeSomaticReadCounts -> JES, case_gatk_acnv_workflow.PlotSegmentedCopyRatio -> JES, case_gatk_acnv_workflow.TumorCorrectGCBias -> JES, case_gatk_acnv_workflow.HetPulldown -> JES, case_gatk_acnv_workflow.PadTargets -> JES
[2016-10-27 13:10:07,81] [info] JES [6f995b2d]: Creating authentication file for workflow 6f995b2d-cf39-4be1-adfb-b6d0a961bd9c at
 gs://my-cromwell-workflows-bucket/case_gatk_acnv_workflow/6f995b2d-cf39-4be1-adfb-b6d0a961bd9c/6f995b2d-cf39-4be1-adfb-b6d0a961bd9c_auth.json
[2016-10-27 13:10:08,17] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:12,13] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
^C[2016-10-27 13:10:13,93] [info] WorkflowManagerActor: Received shutdown signal.
[2016-10-27 13:10:13,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:13,93] [info] WorkflowManagerActor Aborting all workflows
[2016-10-27 13:10:14,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:15,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:16,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:17,93] [info] Waiting for 1 workflows to abort...
^C^C[2016-10-27 13:10:18,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:19,33] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:19,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:20,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:21,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:22,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:23,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:24,84] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:24,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:25,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:26,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:27,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:28,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:29,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:30,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:31,53] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:31,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:32,93] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:33,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:34,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:35,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:36,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:37,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:38,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:39,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:40,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:41,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:42,05] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:42,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:43,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:44,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:45,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:46,38] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:46,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:47,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:48,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:49,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:50,94] [info] Waiting for 1 workflows to abort...
^C[2016-10-27 13:10:51,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:52,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:53,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:54,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:55,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:56,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:57,16] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:10:57,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:58,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:10:59,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:00,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:01,94] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:02,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:03,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:04,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:05,57] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:11:05,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:06,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:07,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:08,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:09,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:10,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:11,95] [info] Waiting for 1 workflows to abort...
Killed
lichtens@lichtens-big:~/test_eval$ [2016-10-27 13:11:12,80] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not Found",
    "reason" : "notFound"
  } ],
  "message" : "Not Found"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-27 13:11:12,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:13,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:14,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:15,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:16,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:17,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:18,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:19,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:20,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:21,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:22,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:23,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:24,95] [info] Waiting for 1 workflows to abort...
[2016-10-27 13:11:25,41] [error] Exception not convertible into handled response
...snip....

The text was updated successfully, but these errors were encountered:

LeeTL1220 · 2016-10-27T13:24:35Z

This one is different from previous JES issues that were error 500.

So this issue might be a dupe.

LeeTL1220 · 2016-10-27T13:48:22Z

Here is another one (only appeared once), but the workflow keeps going... I am not even sure if I need to quit...

[2016-10-27 13:47:19,26] [info] JesAsyncBackendJobExecutionActor [fd2fcb78case_gatk_acnv_workflow.HetPulldown:13:1]: JesAsyncBackendJobExecutionActor [fd2fcb78:case_gatk_acnv_workflow.HetPulldown:13:1] Status change from - to Initializing
[2016-10-27 13:47:19,26] [info] JesAsyncBackendJobExecutionActor [fd2fcb78case_gatk_acnv_workflow.HetPulldown:9:1]: JesAsyncBackendJobExecutionActor [fd2fcb78:case_gatk_acnv_workflow.HetPulldown:9:1] Status change from - to Initializing
[2016-10-27 13:47:19,27] [info] JesAsyncBackendJobExecutionActor [fd2fcb78case_gatk_acnv_workflow.HetPulldown:15:1]: JesAsyncBackendJobExecutionActor [fd2fcb78:case_gatk_acnv_workflow.HetPulldown:15:1] Status change from - to Initializing
[2016-10-27 13:47:24,90] [error] Exception not convertible into handled response
com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
{
  "code" : 503,
  "errors" : [ {
    "domain" : "global",
    "message" : "Backend Error",
    "reason" : "backendError"
  } ],
  "message" : "Backend Error"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

LeeTL1220 · 2016-10-28T14:43:05Z

Here is another error that a user will have trouble determining if the whole workflow failed:

[2016-10-28 14:37:09,70] [warn] 1 failures fetching JES statuses: {"domain":"global","message":"Deadline expired before operation could complete.","reason":"backendError"}
[2016-10-28 14:37:09,70] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.CNLoHAndSplitsCaller:2:1]: Caught exception, retrying:
java.io.IOException: Google request failed: {
  "code" : 504,
  "errors" : [ {
    "domain" : "global",
    "message" : "Deadline expired before operation could complete.",
    "reason" : "backendError"
  } ],
  "message" : "Deadline expired before operation could complete.",
  "status" : "DEADLINE_EXCEEDED"
}
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:30)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:37:35,25] [error] Read timed out
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
        at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:241)
        at cromwell.backend.impl.jes.statuspolling.JesPollingActor.runBatch(JesPollingActor.scala:67)
        at cromwell.backend.impl.jes.statuspolling.JesPollingActor.cromwell$backend$impl$jes$statuspolling$JesPollingActor$$handleBatch(JesPollingActor.scala:58)
        at cromwell.backend.impl.jes.statuspolling.JesPollingActor$$anonfun$receive$1.applyOrElse(JesPollingActor.scala:36)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.statuspolling.JesPollingActor.aroundReceive(JesPollingActor.scala:22)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2016-10-28 14:37:35,25] [error] The JES polling actor Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$c/$a/$a#551868791] unexpectedly terminated while conducting 5 polls. Making a new one...
[2016-10-28 14:37:35,25] [info] watching Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$c/$a/$b#797880880]
[2016-10-28 14:37:35,25] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.CNLoHAndSplitsCaller:1:1]: Caught exception, retrying:
java.lang.RuntimeException: Unexpected actor death!
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:37:35,25] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.AllelicCNV:10:1]: Caught exception, retrying:
java.lang.RuntimeException: Unexpected actor death!
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:37:35,25] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.PlotACNVResults:6:1]: Caught exception, retrying:
java.lang.RuntimeException: Unexpected actor death!
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:37:35,25] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.PlotACNVResults:1:1]: Caught exception, retrying:
java.lang.RuntimeException: Unexpected actor death!
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:37:35,25] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.CNLoHAndSplitsCaller:6:1]: Caught exception, retrying:
java.lang.RuntimeException: Unexpected actor death!
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:38:11,89] [warn] 1 failures fetching JES statuses: {"domain":"global","message":"Deadline expired before operation could complete.","reason":"backendError"}
[2016-10-28 14:38:11,89] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.AllelicCNV:7:1]: Caught exception, retrying:
java.io.IOException: Google request failed: {
  "code" : 504,
  "errors" : [ {
    "domain" : "global",
    "message" : "Deadline expired before operation could complete.",
    "reason" : "backendError"
  } ],
  "message" : "Deadline expired before operation could complete.",
  "status" : "DEADLINE_EXCEEDED"
}
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:30)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2016-10-28 14:38:17,54] [info] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.AllelicCNV:3:1]: JesAsyncBackendJobExecutionActor [a3dd8163:case_gatk_acnv_workflow.AllelicCNV:3:1] Status change from Running to Success
[2016-10-28 14:38:17,67] [info] WorkflowExecutionActor-a3dd8163-de37-4467-a227-5364959a8940 [a3dd8163]: Starting calls: case_gatk_acnv_workflow.CNLoHAndSplitsCaller:3:1, case_gatk_acnv_workflow.PlotACNVResults:3:1
[2016-10-28 14:38:18,14] [info] JesRun [a3dd8163case_gatk_acnv_workflow.CNLoHAndSplitsCaller:3:1]: JES Run ID is operations/ENPH6N2AKxi-zoCK0M65gEAgn5eRl70GKg9wcm9kdWN0aW9uUXVldWU
[2016-10-28 14:38:18,31] [info] JesRun [a3dd8163case_gatk_acnv_workflow.PlotACNVResults:3:1]: JES Run ID is operations/EPfI6N2AKxi_iI64ku3M2xAgn5eRl70GKg9wcm9kdWN0aW9uUXVldWU
[2016-10-28 14:38:43,07] [info] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.CNLoHAndSplitsCaller:3:1]: JesAsyncBackendJobExecutionActor [a3dd8163:case_gatk_acnv_workflow.CNLoHAndSplitsCaller:3:1] Status change from - to Initializing
[2016-10-28 14:38:43,07] [info] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.PlotACNVResults:3:1]: JesAsyncBackendJobExecutionActor [a3dd8163:case_gatk_acnv_workflow.PlotACNVResults:3:1] Status change from - to Initializing
[2016-10-28 14:38:43,07] [warn] 1 failures fetching JES statuses: {"domain":"global","message":"Deadline expired before operation could complete.","reason":"backendError"}
[2016-10-28 14:38:43,07] [warn] JesAsyncBackendJobExecutionActor [a3dd8163case_gatk_acnv_workflow.AllelicCNV:4:1]: Caught exception, retrying:
java.io.IOException: Google request failed: {
  "code" : 504,
  "errors" : [ {
    "domain" : "global",
    "message" : "Deadline expired before operation could complete.",
    "reason" : "backendError"
  } ],
  "message" : "Deadline expired before operation could complete.",
  "status" : "DEADLINE_EXCEEDED"
}
        at cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:30)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

katevoss · 2017-04-04T17:03:56Z

@LeeTL1220 does this still happen?

LeeTL1220 · 2017-04-04T17:25:46Z

Yes. Not as bad, but still pretty hard to debug. A fair number of transient errors when I use JES.

…

On Tue, Apr 4, 2017 at 1:03 PM, Kate Voss ***@***.***> wrote: @LeeTL1220 <https://github.com/LeeTL1220> does this still happen? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACDXk-_InNBhbZ4MPXcOdseOkaYuW2N2ks5rsnf_gaJpZM4KiVST> .

-- Lee Lichtenstein Broad Institute 75 Ames Street, Room 7003EB Cambridge, MA 02142 617 714 8632

katevoss · 2017-04-04T18:26:19Z

@LeeTL1220 we have made number of improvements with transient errors in JES, Cromwell (in version 26) will now retry transient errors automatically.

LeeTL1220 changed the title ~~JES error causes cromwell to hang and spew useless error messages~~ JES error causes cromwell to hang and spew useless (for a user) error messages Oct 27, 2016

LeeTL1220 mentioned this issue Nov 1, 2016

JES errors are hard to debug and better reporting is helpful #1637

Closed

katevoss closed this as completed Apr 4, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JES error causes cromwell to hang and spew useless (for a user) error messages #1631

JES error causes cromwell to hang and spew useless (for a user) error messages #1631

LeeTL1220 commented Oct 27, 2016

LeeTL1220 commented Oct 27, 2016

LeeTL1220 commented Oct 27, 2016 •

edited

LeeTL1220 commented Oct 28, 2016

katevoss commented Apr 4, 2017

LeeTL1220 commented Apr 4, 2017 via email

katevoss commented Apr 4, 2017

JES error causes cromwell to hang and spew useless (for a user) error messages #1631

JES error causes cromwell to hang and spew useless (for a user) error messages #1631

Comments

LeeTL1220 commented Oct 27, 2016

LeeTL1220 commented Oct 27, 2016

LeeTL1220 commented Oct 27, 2016 • edited

LeeTL1220 commented Oct 28, 2016

katevoss commented Apr 4, 2017

LeeTL1220 commented Apr 4, 2017 via email

katevoss commented Apr 4, 2017

LeeTL1220 commented Oct 27, 2016 •

edited