TES Backend Prototype. Closes #1294 #1816

adamstruck · 2017-01-05T22:36:05Z

This is a prototype backend that communicates via the GA4GH task execution schema. The reference implementation of the task server that we have been developing against is here.

This is a work in progress and definitely could use some feedback from you with regards to how to better fit this in with your current code base and future development plans.

In the coming weeks we plan to implement unit tests, add documentation and refactor the code based on your recommendations.

* Pretty hacky in almost every way. Just committing a checkpoint.

…hanae Merge tes-backend-buchanae

kshakir · 2017-01-31T14:32:21Z

@adamstruck I still need to do a more in depth review if you're looking for scala syntax feedback (ex: case … => { … } could be case … => …).

Early feedback:

PR 1930 contains a few more changes to the standard backend api. I tested cherry-picking 1930 onto this PR to see what would be left to patch up. Using an "Obsolete" set of bridge code for now, these are the minimal changes for the updated path api, plus changes for standardized command line generation. NOTE: 1930 is still under review and may change, plus the linked github commit will be deleted once these two PRs are merged.
The standard backend will continue to change for a while as we move more common code. For example, the script generation for globs is now centralized as of PR 1930. The only CI testing I am aware of at the moment is sbt tesBackend/test that runs under travis. Is there a dockerized solution yet for TES that we could use with travis centaur, like we have for JES and Local? Otherwise, the minimal patches above pass the very, very basic sbt test unit tests.
Your PR is ok as is, but I need to think about necessity of Async.await a bit more. The standard backend api is synchronous, requiring the Async.await. But the underlying "basic" backend trait is using scala futures, and I need more insight into how those are interacting with the akka actors. For example, I wouldn't want the actors receiving multiple akka poll messages in the mailbox and then queuing up dozens of overlapping poll futures in the thread pool. I also really like that your awaits have timeouts and aren't infinite futures.

More to come. Thanks!

kshakir

Various update requests. Hopefully nothing major. Let me know if there are any questions.

kshakir · 2017-02-01T15:49:45Z

core/src/main/resources/reference.conf

@@ -157,7 +157,7 @@ engine {
 }

 backend {
-  default = "Local"
+  default = "TES"


OK for debugging locally, but will likely break even Travis tests. One should probably debug with a local tes.conf and java -Dconfig.file=tes.conf instead, or sbt -J-Dconfig.file=tes.conf.

kshakir · 2017-02-01T15:52:37Z

core/src/main/resources/reference.conf

+      endpoint = "http://127.0.0.1:8000/v1/jobs"
+      poll-interval = 10
+    }
+    }


This is changing soon, but for now, document your backend config as a commented out string here in the reference.conf.

kshakir · 2017-02-01T15:54:55Z

...Backends/tes/src/main/scala/cromwell/backend/impl/tes/TesAsyncBackendJobExecutionActor.scala

+import cromwell.core.path.{PathBuilder, PathFactory}
+import cromwell.core.retry.SimpleExponentialBackoff
+import lenthall.util.TryUtil
+import spray.client.pipelining._


We're moving away from spray to akka-http as the former has been end-of-lifed. Can you switch over your API?

To throw another wrench in the mix, another idea is to use a library generated from the protobuf/grpc task execution schemas. Seems like possibly everyone is missing out on one of the primary benefits of using protobuf and grpc by implementing everything as http/json.

kshakir · 2017-02-02T16:37:39Z

...Backends/tes/src/main/scala/cromwell/backend/impl/tes/TesAsyncBackendJobExecutionActor.scala

+    val submitTask = pipeline[TesPostResponse]
+      .apply(Post(tesEndpoint, taskMessage))
+
+    val response = Await.result(submitTask, 15.seconds)


Mea culpa. I filed ticket #1218 to address the issue of using scala futures instead of akka actors in the backend. In standard, I tried to cut down of the spread of futures until I had more time to look at the issue. After further research, attempting to temporarily reduce the use of scala futures by using synchronous interfaces in the standard backend was a step in the wrong direction. [1] [2]

I submitted a PR #1947 for the standard backend that allows execute and recover to also go back to using scala futures, in addition to the current pollStatusAsync. Instead of using Await.result, you should override the async versions of the methods instead, and compose your Future result as necessary.

If you still want to shortcut your actions after some timeout, there are other ways of providing timeouts with scala futures. [3] But I suspect your spray client will have some timeouts already included.

Eventually I hope to be able to adapt the backend api's use of futures with proper akka actors, as has already begun with the JES batch polling actor. That way one could maybe use akka messages instead to timeout, have akka mailboxes queue messages instead of directly queuing multiple futures onto the thread pools, etc.

TL;DR: Instead of using Await.result, return a Future by overriding pollStatusAsync, and the soon to come executeAsync.

kshakir · 2017-02-02T16:45:13Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesJobPaths.scala

+        val localPath = Paths.get(file.valueString).toAbsolutePath
+        val containerPath = containerInput(localPath.toString)
+        WdlFile(containerPath)
+      }


Remove the {}, creating the extra block under the case statement.

case file: WdlFile => val localPath = Paths.get(file.valueString).toAbsolutePath val containerPath = containerInput(localPath.toString) WdlFile(containerPath)

kshakir · 2017-02-02T22:04:42Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

+        tesPaths.storageOutput(f),
+        tesPaths.containerOutput(f),
+        "File",
+        Some(false)


Instead of Some() use Option() (and above on line 73)

kshakir · 2017-02-02T22:05:31Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

+            tesPaths.storageOutput(outputFile),
+            tesPaths.containerOutput(outputFile),
+            "File",
+            Some(false)


Instead of Some() use Option() (and above on line 99)

kshakir · 2017-02-02T22:05:56Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

+            tesPaths.storageOutput(globListFile),
+            tesPaths.containerOutput(globListFile),
+            "File",
+            Some(false)


Instead of Some() use Option() (and above on lines 116, 120, and 124)

kshakir · 2017-02-02T22:06:23Z

supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

+    runtimeAttributes.cpu,
+    runtimeAttributes.memory.to(MemoryUnit.GB).amount.toInt,
+    Some(false),
+    Some(volumes),


Instead of Some() use Option() (and above on lines 141, 149, and 158)

Just for my own personal Scala education, could you explain this one in more detail? volumes is definitely I non-null value I think, so isn't that Option() just converted to Some()? Is this a style thing or is there a case where one is better than the other?

I think I learned Some usage from articles like this, http://alvinalexander.com/scala/best-practice-option-some-none-pattern-scala-idioms

This is something we picked up from my previous team where nulls weren't verboten and we'd get bit by this now and then.Option(null) is None whereas Some(null) is exactly what it says.

Generally my preference is to avoid things which the developer "knows" to be true in favor of things which must be true by definition as the latter doesn't require any further thinking on a future developer - also the latter might end up becoming untrue w/o people realizing.

Another example for me is .get on an Option, sure there are times where I know that it is Some and won't blow up but I'd much prefer to not do that for the reasons cited above. That said even I will drop my moral stance at times due to the pain in the ass factor :)

Excellent explanation. Thanks @geoffjentry ! Completely agree with your reasoning.

I should also toss in why do I still say this for examples like Some(false) where it can't possibly be null. There it's more about building up the muscle memory and the uniformity of it. Admittedly I care less in those situations.

kshakir · 2017-02-02T22:06:45Z

supportedBackends/tes/src/test/scala/cromwell/backend/impl/tes/TesRuntimeAttributesSpec.scala

+
+    "validate a valid dockerWorkingDir entry" in {
+      val runtimeAttributes = Map("docker" -> WdlString("ubuntu:latest"), "dockerWorkingDir" -> WdlString("/tmp"))
+      val expectedRuntimeAttributes = expectedDefaults.copy(dockerWorkingDir = Some("/tmp"))


Instead of Some() use Option()

Standard Backend has even more default implementations, and a few breaking changes. Moved some code for testing JES out of main. Increased the timeout for the ask inside SingleWorkflowRunnerActorSpec. Bumped apache http client to the latest version.

Updated path builder usage and added tests. NOTE: GcsPathBuilder.build().get.toString now returns "gs://bucket/path" instead of "/path".

Changed def's with concrete implementations to lazy vals. Reverted lines that have not changed since develop. Minor updates to comments.

adamstruck · 2017-02-05T22:08:21Z

@kshakir thanks for your review. I will make the changes you requested during the coming week.

…actor

Refactor to address most of the PR comments. TODO: - migrate from spray to akka http - set up centuar build for testing

kshakir · 2017-02-08T05:51:20Z

FYI- the failing test suite "centaurJes" is due to a know limitation in our test setup, but the other three look good, including the "centaurLocal" tests. Again, without a dockerized "centaurTes" we'll certainly try to avoid issues, but there won't be any guarantees that upgrades to the standard backend API don't break TES.

Also, the 87 commits will need to be rebased and squashed correctly to a minimal set, on the order of 1 commit, but otherwise let us know when you're ready for re-review.

Let us know if you have more questions or if we can provide any other assistance.

adamstruck · 2017-02-08T19:20:32Z

I am working on getting centuar testing up and running with a TES implementation. Rather than going through rebase hell on this branch (I have numerous merge commits), I will submit a new PR from another branch where I have squashed the commits via merge. How does that sound?

adamstruck · 2017-02-13T18:56:24Z

Tracking on #1979

adamstruck and others added 30 commits October 17, 2016 11:39

inital commit of tes backend

09262af

compile error fixes

d7cb75a

spray json stuff

dd3f939

...

4f67026

Get spray json working

976debd

broken spray client

2119ab0

blah

2c91ab5

broken.. compile errors

ae4bb97

foo

2aa9996

foo

fc5742e

compiles and runs

056f6a9

fo

34109c8

slight cleanup

19240ca

Doesn't quite work but starting to pull stuff out of the WDL

117ec58

works against new PR

a794c73

WIP: instantiateCommandLine and input TaskParameters

53501bf

* Pretty hacky in almost every way. Just committing a checkpoint.

Clean up command function and fix TesTask call.

9d39b05

Raise error when "docker" attribute is missing

a53f43c

Refactor TesTask and TesTaskMessage

e068f3c

minor progress

0aff439

Minor refactoring

1efd8ae

Merge branch 'ga4gh_hackathon_tes' into tes-backend-buchanae

b61e980

Clean up compile errors after code merge

0a90ae6

fixed toDockerPath; configurable DockerRoot

5359895

Merge pull request #1 from ohsu-computational-biology/tes-backend-buc…

9241e7f

…hanae Merge tes-backend-buchanae

Merge remote-tracking branch 'origin/ga4gh_hackathon_tes' into as_tes

d8beff8

handled stdout and stderr in DockerExecuter

8c055de

removed dockerOutputDir

16c547e

Working on TesTask outputs and other minor refactoring

ab965f7

Refactoring paths and working on output paths

96af2df

adamstruck added 2 commits January 30, 2017 15:32

Merge remote-tracking branch 'upstream/develop' into tes_backend

a78d3b6

fixed tes initialization actor spec

f9c834e

cjllanwarne changed the title ~~TES Backend Prototype~~ TES Backend Prototype. Closes #1294 Jan 31, 2017

Merge remote-tracking branch 'upstream/develop' into tes_backend

8a897ef

kshakir requested changes Feb 2, 2017

View reviewed changes

kshakir added 8 commits February 3, 2017 21:04

Moved globbing behavior to standard.

9ff6291

Standard Backend has even more default implementations, and a few breaking changes. Moved some code for testing JES out of main. Increased the timeout for the ask inside SingleWorkflowRunnerActorSpec. Bumped apache http client to the latest version.

Wrapped java nio paths with our own paths.

e565670

Updated path builder usage and added tests. NOTE: GcsPathBuilder.build().get.toString now returns "gs://bucket/path" instead of "/path".

PR fixup

306c95f

Bug fix: Don't try to resolve gcs paths on top of default paths.

123baf9

Added note on file:/// to change log.

eb6f893

Map wdl pairs and optionals in addition to maps and arrays.

ae2a0e8

Revert errant changes.

d6f2819

Various cleanup.

40bfc4f

Changed def's with concrete implementations to lazy vals. Reverted lines that have not changed since develop. Minor updates to comments.

adamstruck added 9 commits February 5, 2017 14:11

Merge remote-tracking branch 'upstream/develop' into tes_backend

60ecfc1

Merge remote-tracking branch 'upstream/ks_glob_standard' into tes_ref…

2719792

…actor

changes from kshakir/ks_tes_backend_plus_some_glob_standard

d0c93db

Merge remote-tracking branch 'upstream/develop' into tes_refactor

3f311e7

formatting and other changes from PR feedback

6432bf5

fixed output processing

de39e03

revert reference.conf

fb51be3

added concurrent-job-limit

83dff68

Merge pull request #4 from ohsu-comp-bio/tes_refactor

8819061

Refactor to address most of the PR comments. TODO: - migrate from spray to akka http - set up centuar build for testing

adamstruck mentioned this pull request Feb 13, 2017

TES Backend Prototype. Closes #1294 #1979

Merged

adamstruck closed this Feb 13, 2017

adamstruck deleted the tes_backend branch February 22, 2017 23:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TES Backend Prototype. Closes #1294 #1816

TES Backend Prototype. Closes #1294 #1816

adamstruck commented Jan 5, 2017

kshakir commented Jan 31, 2017

kshakir left a comment

kshakir Feb 1, 2017

kshakir Feb 1, 2017

kshakir Feb 1, 2017

buchanae Feb 3, 2017

kshakir Feb 2, 2017

kshakir Feb 2, 2017

kshakir Feb 2, 2017

kshakir Feb 2, 2017

kshakir Feb 2, 2017

kshakir Feb 2, 2017

buchanae Feb 3, 2017

geoffjentry Feb 3, 2017

buchanae Feb 3, 2017

geoffjentry Feb 3, 2017

kshakir Feb 2, 2017

adamstruck commented Feb 5, 2017

kshakir commented Feb 8, 2017

adamstruck commented Feb 8, 2017

adamstruck commented Feb 13, 2017

TES Backend Prototype. Closes #1294 #1816

TES Backend Prototype. Closes #1294 #1816

Conversation

adamstruck commented Jan 5, 2017

kshakir commented Jan 31, 2017

kshakir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamstruck commented Feb 5, 2017

kshakir commented Feb 8, 2017

adamstruck commented Feb 8, 2017

adamstruck commented Feb 13, 2017