Confirm shards are running in parallel

[tir38]

I initially was concerned about whether shards were run parallel or not, but didn't have good logs to confirm, So I raised #577

Now that we have good logs (thanks BTW) I still don't think our shards are run in parallel (despite what I said here )

Here are our most recent logs: https://gist.github.com/tir38/ee2bf14148e7d7c14aa2be48c59a2de1

We run 5 devices and 5 shards

We are seeing each shard take ~4-5 minutes

  Shard 0: Expected: 269s, Actual: 265s, Diff: -4s
  Shard 1: Expected: 269s, Actual: 273s, Diff: 3s
  Shard 2: Expected: 270s, Actual: 259s, Diff: -11s
  Shard 3: Expected: 272s, Actual: 274s, Diff: 2s
  Shard 4: Expected: 272s, Actual: 266s, Diff: -6s

But the total task time is closer to 20 minutes:

BUILD SUCCESSFUL in 20m 33s

I know @bootstraponline said before that "running multiple matrices to enable parallel execution isn't ideal" Is it possible that FTL is delaying each shard?

[bootstraponline]

This will be solved soon by server side sharding. FTL implemented this and I've done work to prototype what it'd look like in Flank. Unfortunately TestArmada has reduced permissions of non-Walmart employees. I expect on Jan 2 we'll get approval from Walmart to transfer Flank into the Flank GitHub org and then feature development will resume.

My guess is that FTL is queueing some of your executions which don't show up in the execution time but will result in clock wall time.

[tir38]

sweet! I saw that https://issuetracker.google.com/issues/115928363#comment4 Looking forward to it.

[bootstraponline]

Hey, server side sharding is implemented on master. We also have a shiny new Flank org on GitHub. 🙂

[runningcode]

I'm seeing this as well. In which version is this fixed?

[bootstraponline]

this should be fixed in the snapshot release

[tir38]

We've been running with this for a month or more. Very nice work y'all. Thanks a million for the hard work!

[tasomaniac]

We are using the Flank version: v21.03.1 but we are still seeing shards running sequentially. Is there anything special that needs to be done? I couldn't find in the docs.