TES Backend Prototype. Closes #1294 #1979

adamstruck · 2017-02-13T18:55:54Z

Picking up from #1816.

Setup for Centaur testing. The current TES schema doesn't allow for changing the default user, so centaur fails on:

non_root_specified_user

Is there a way to configure Centaur to skip specific tests?

kshakir · 2017-02-13T22:25:36Z

Try sbt 'testOnly -- -l localdockertest'

kshakir · 2017-02-13T22:34:40Z

src/bin/travis/testCentaurTes.sh

+# because pulling the image during some of the tests would cause them to fail 
+# (specifically output_redirection which expects a specific value in stderr)
+docker pull ubuntu:latest
+centaur/test_cromwell.sh -j"${CROMWELL_JAR}" -c${TES_CENTAUR_CONF}


Until we figure out excluding these local tests by default, change:

centaur/test_cromwell.sh -j"${CROMWELL_JAR}" -c${TES_CENTAUR_CONF}

to:

centaur/test_cromwell.sh -j"${CROMWELL_JAR}" -c${TES_CENTAUR_CONF} -elocaldockertest

adamstruck · 2017-02-14T17:42:38Z

@kshakir adding -elocaldockertest addressed the issue when I ran the tests locally. In Travis CI, the non_root_default_user test is failing with:

Status: Downloaded newer image for mcovarr/notroot:v1
bin/bash: /cromwell-executions/woot/148f812b-028b-4264-82bd-ab2f089efe98/call-notroot/execution/script: Permission denied

This test passes when I run it locally. Any ideas?

mcovarr · 2017-02-14T21:55:17Z

@adamstruck That test tries to run a Docker image which has a default user which is not root (the test was created based on this ticket). That error looks like the non-root user in the container doesn't have execute permission on the script that was created by Cromwell running as a different user.

kshakir · 2017-02-15T00:36:44Z

...Backends/tes/src/main/scala/cromwell/backend/impl/tes/TesAsyncBackendJobExecutionActor.scala

+  def createTaskMessage(): TesTaskMessage = {
+    val task = TesTask(jobDescriptor, configurationDescriptor, jobLogger, tesJobPaths, runtimeAttributes, commandDirectory, backendEngineFunctions)
+
+    tesJobPaths.script.write(commandScriptContents)


Perhaps, as Miguel pointed out, changing the permissions might help, specifically here?

tesJobPaths.script.write(commandScriptContents).chmod("rwxrwxrwx")

I haven't tried running this myself yet though. I'm assuming that this is the right spot to fix permissions, based on the error message in the logs producing the exit code 126:

/bin/bash: /cromwell-executions/woot/a2c488a2-e2a0-46f4-8714-47ac3a89bc2b/call-notroot/execution/script: Permission denied

I'm not 100% sure at the moment where/why the permissions on the script are not-as-expected though. The local backend gets around permissions errors by telling docker to run /bin/bash /path/to/script, that seems to work even if the file is rw-rw-r--.

If you really are running into problems and don't want to support this test and the "running as non-root" functionality for whatever reason, let us know, and we can figure out a way to exclude it.

This was an issue on the TES side of things. I am resolving this today.

adamstruck · 2017-02-16T19:02:07Z

@kshakir all TES Centaur tests are now passing.

kshakir · 2017-02-17T18:26:23Z

src/bin/travis/testCentaurTes.sh

+cd ..
+
+TES_CONF="$(pwd)/src/bin/travis/resources/tes.conf"
+git clone https://github.com/ohsu-comp-bio/funnel.git


@adamstruck Can you change this to https://github.com/broadinstitute/funnel.git? I've forked the repo.

kshakir · 2017-02-17T18:30:58Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesBackendFileHashing.scala

+    } get
+
+    tryWithResource(() => path.newInputStream) { inputStream =>
+      org.apache.commons.codec.digest.DigestUtils.md5Hex(inputStream)


ToL: Note for the future, this is fine for passing centaur tests, but takes too long on real BAM files. Hopefully whenever #1843 is implemented it will be reusable by this backend also.

kshakir · 2017-02-17T18:34:08Z

👍 once centaur isn't downloading from an external repo

mcovarr · 2017-02-17T18:36:54Z

README.md

@@ -75,7 +80,8 @@ A [Workflow Management System](https://en.wikipedia.org/wiki/Workflow_management
  * [Local Filesystem Options](#local-filesystem-options)
 * [Imports](#imports)
 * [Sub Workflows](#sub-workflows)
-* [Meta blocks](#meta-blocks)
+  * [Execution](#execution)
+  * [Metadata](#metadata)


oops thanks for fixing this!

mcovarr · 2017-02-17T18:52:44Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesRuntimeAttributes.scala

+  private val MemoryDefaultValue = "2 GB"
+
+  val DiskSizeKey = "disk"
+  private val DiskSizeDefaultValue = "2 GB"


These default values are not consistent with the README

mcovarr · 2017-02-17T19:06:37Z

README.md

@@ -1392,6 +1399,65 @@ In order to monitor metrics (CPU, Memory, Disk usage...) about the VM during Cal

 The output of this script will be written to a `monitoring.log` file that will be available in the call gcs bucket when the call completes.  This feature is meant to run a script in the background during long-running processes.  It's possible that if the task is very short that the log file does not flush before de-localization happens and you will end up with a zero byte file.

+## GA4GH TES Backend
+The TES backend submit jobs to a server that complies with the protocol described by the [GA4GH schema](https://github.com/ga4gh/task-execution-schemas).


submit jobs => submits jobs

mcovarr · 2017-02-17T19:27:34Z

👍

geoffjentry · 2017-02-17T20:48:15Z

Pretty cool, thanks @adamstruck

adamstruck · 2017-02-17T21:42:47Z

Thanks for the comments. I pushed the README changes and Funnel is now being downloaded from the broad's fork.

It looks like both the Local and Tes Centaur build are failing now though?

added -elocaldockertest to centaur command print TES logs after run fixed unmarshalling bug inputs in read-only volume; matched TES case classes to schema address final PR comments

kshakir · 2017-02-18T02:34:29Z

The Local build did fail, due to an unrelated hiccup. A restart of the build cleared the error.

Meanwhile, JES Centaur failed due to #1717, but TES passed just fine.

Everything looking great. Thanks again for all of your contributions!

adamstruck mentioned this pull request Feb 13, 2017

TES Backend Prototype. Closes #1294 #1816

Closed

kshakir reviewed Feb 13, 2017

View reviewed changes

adamstruck force-pushed the tes_backend_prototype branch 3 times, most recently from 1a87c5e to 0e02f81 Compare February 14, 2017 19:01

kshakir reviewed Feb 15, 2017

View reviewed changes

adamstruck force-pushed the tes_backend_prototype branch from 01c2e2e to b9faa69 Compare February 16, 2017 18:46

kshakir reviewed Feb 17, 2017

View reviewed changes

kshakir assigned mcovarr Feb 17, 2017

mcovarr approved these changes Feb 17, 2017

View reviewed changes

adamstruck force-pushed the tes_backend_prototype branch from b9faa69 to 97b2116 Compare February 17, 2017 20:53

tes backend prototype with centuar setup

d7adb19

added -elocaldockertest to centaur command print TES logs after run fixed unmarshalling bug inputs in read-only volume; matched TES case classes to schema address final PR comments

adamstruck force-pushed the tes_backend_prototype branch from 97b2116 to d7adb19 Compare February 17, 2017 22:14

kshakir merged commit 1524f3d into broadinstitute:develop Feb 18, 2017

adamstruck deleted the tes_backend_prototype branch May 12, 2017 21:45

aednichols mentioned this pull request Mar 28, 2024

WX-1535 Change Docker root on Azure to match GCP #7395

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TES Backend Prototype. Closes #1294 #1979

TES Backend Prototype. Closes #1294 #1979

adamstruck commented Feb 13, 2017

kshakir commented Feb 13, 2017

kshakir Feb 13, 2017

adamstruck commented Feb 14, 2017 •

edited

mcovarr commented Feb 14, 2017

kshakir Feb 15, 2017

adamstruck Feb 16, 2017

adamstruck commented Feb 16, 2017

kshakir Feb 17, 2017

kshakir Feb 17, 2017

kshakir commented Feb 17, 2017 •

edited by briandmaher

mcovarr Feb 17, 2017

mcovarr Feb 17, 2017

mcovarr Feb 17, 2017

mcovarr commented Feb 17, 2017 •

edited by briandmaher

geoffjentry commented Feb 17, 2017

adamstruck commented Feb 17, 2017

kshakir commented Feb 18, 2017

TES Backend Prototype. Closes #1294 #1979

TES Backend Prototype. Closes #1294 #1979

Conversation

adamstruck commented Feb 13, 2017

kshakir commented Feb 13, 2017

Choose a reason for hiding this comment

adamstruck commented Feb 14, 2017 • edited

mcovarr commented Feb 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamstruck commented Feb 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kshakir commented Feb 17, 2017 • edited by briandmaher

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcovarr commented Feb 17, 2017 • edited by briandmaher

geoffjentry commented Feb 17, 2017

adamstruck commented Feb 17, 2017

kshakir commented Feb 18, 2017

adamstruck commented Feb 14, 2017 •

edited

kshakir commented Feb 17, 2017 •

edited by briandmaher

mcovarr commented Feb 17, 2017 •

edited by briandmaher