Fixed issue regarding getting request body on query POST API.

[francares]

There is a need to integrate query API with some of the components I'm developing and while I was testing with different versions of Cromwell (0.24, 0.28 and 0.29) I found out the following:

• 0.24/0.28: when Cromwell is under load and trying to query +100 workflows, Cromwell (Slick) throws stack overflow exception.
• 0.29: seems to work fine but I found that query POST API was returning everything since query params where empty and request body was parsed.

This is the fix that worked for me.

[geoffjentry]

@francares thanks. for the earlier versions you should be able to fix that by tweaking db-batch-size up higher (try 200 or 500 and see how it goes) in the metadata service conf. Although now that I say it IIRC you all are using a metadata service other than the default? If so this won't help.

The 29 one is just a bug on my part, thanks

[geoffjentry]

👍

[mcovarr]

@francares are you seeing a stack overflow exception or a Slick "task rejected from queue" exception?

[geoffjentry]

@mcovarr oh good catch. Further evidence that perhaps this is something different altogether

[francares]

Uncaught error from thread [cromwell-system-akka.dispatchers.api-dispatcher-57] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.StackOverflowError
	at scala.collection.immutable.Set$EmptySet$.seq(Set.scala:68)
	at scala.collection.SetLike$class.$plus$plus(SetLike.scala:141)
	at scala.collection.AbstractSet.$plus$plus(Set.scala:47)
	at slick.compiler.ExpandSums.slick$compiler$ExpandSums$$tr$1(ExpandSums.scala:27)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.util.ConstArray.endoMap(ConstArray.scala:122)
	at slick.ast.Node$class.mapChildren(Node.scala:51)
	at slick.ast.Apply.mapChildren(Node.scala:547)
	at slick.compiler.ExpandSums.slick$compiler$ExpandSums$$tr$1(ExpandSums.scala:32)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.util.ConstArray.endoMap(ConstArray.scala:122)
	at slick.ast.Node$class.mapChildren(Node.scala:51)
	at slick.ast.Apply.mapChildren(Node.scala:547)
	at slick.compiler.ExpandSums.slick$compiler$ExpandSums$$tr$1(ExpandSums.scala:32)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.compiler.ExpandSums$$anonfun$7.apply(ExpandSums.scala:32)
	at slick.util.ConstArray.endoMap(ConstArray.scala:122)
	at slick.ast.Node$class.mapChildren(Node.scala:51)
	at slick.ast.Apply.mapChildren(Node.scala:547)
	...

It's easy to reproduce, just run +100 hello world workflows and then query for all those workflows ids using query POST API.

[mcovarr]

👍

[geoffjentry]

@francares Thanks. I'm a little concerned that that just magically disappeared with 29 and that perhaps the bug fix was bug, not feature. :)

[francares]

@geoffjentry I tried to reproduce with 29 but I was not able with 500 hello world workflows. Anyway, if it happens again I'll let you know...

[geoffjentry]

@francares Cool. My main concern there was that when i did the akka http conversion that I "fixed" it by giving bad results, so as long as the results look good I'll stand down my fretting :)

[mcovarr]

Hmm that is definitely different from the "task rejected from queue" errors. And anyway 28 has the larger default metadata batch size changes, so if this really was a different symptom of that problem it shouldn't be happening on 28.

I don't see much different between develop and 28_hotfix that could legitimately explain fixes in the vicinity of Slick. 😕

[francares]

@mcovarr Yeap, it's weird but I ran +1000 workflows a minute ago and I didn't see issues with 29. I tried the same with 28_hotfix and with the first 100 I got the exception.

[mcovarr]

Thanks @francares, @geoffjentry is going to make a ticket to investigate this on our side.

[geoffjentry]

#2450