Ordering test bundles by test execution times read from an input json file

[ravimandala]

Introducing an configuration to provide a json file with test case level execution times and subsequently use the execution times to order the test bundles from longest to shortest, for better concurrency. \cc @ob
Ref: #336

[ob]

Thanks a lot for picking this up. It'd be awesome if you could add some test cases that demonstrate how this works.

[ravimandala]

I'll incorporate the comments and add a couple of tests.

[ravimandala]

@oliverhu I have incorporated Oscar and Shawn's comments and added some unit tests. Can you please review? Thanks.

[ravimandala]

Same changes are available on #347 but on a different base (tag - v4.1.1).

[ob]

Thanks for addressing the previous review, I left some other comments.

[ob]

Oh, one more thing... where is the timings JSON file supposed to come from? I didn't see changes where Bluepill will emit it.

[ravimandala]

Incorporated your comments. Please review.

[ob]

Looks good. I think the only piece missing is that bluepill should generate the JSON file with the times. After all, it has all the information at the end, when it does the trace-profile.json file...

[ravimandala]

Looks good. I think the only piece missing is that Bluepill should generate the JSON file with the times. After all, it has all the information at the end, when it does the trace-profile.json file...

This is an external input to Bluepill and it is not directly derived from any specific thing that Bluepill emits. Based on my experimentation, times from one single execution are NOT reliable estimates for the next execution. In our case, we calculate medians of execution times of a test over the last 7 days.