Ability to Interleave Performance Runs for Baseline & Test Builds

[piyush286]

Background About Interleaved Runs

While doing the benchmarking comparison between different builds, it's crucial to interleave runs for both baseline and test builds in order to get the most consistent and reliable results.

For benchmarking, we always launch several iterations of a benchmark with a specific build to get performance results for various metrics such as throughput and startup time. These relative numbers are not very useful since they could change when benchmark is run on another platform, when the machine state isn't identical or when the configs are slightly different. Hence, we always use a baseline to gauge the performance of a newer test build.

There are various machine factors that could affect the numbers between multiple iterations, even though they might be running with the same configs. Hence, interleaving runs helps in avoiding those issues and makes sure that the same factor would affect both baseline and test build runs. In order to keep this issue short, I won't get into the benefits and scenarios of interleaved runs.

If T = Test Build; B = Baseline Build, # = Iteration

Interleaving Run Pattern:
Do alternate iteration of each baseline and test build in a ping-pong fashion.
T1, B1, T2, B2, T3, B3

Non-interleaved Run Pattern:
Do all iterations for one build and then do all iterations for another.
T1, T2, T3, B1, B2, B3

Related Issue for Openjdk-tests Framework

adoptium/aqa-tests#850

Background About PerfNext

Currently, PerfNext does not have the capability of launching interleaved runs. It uses non-interleaved runs and launches all iterations under one Jenkins job by using a loop for iterations as shown below:

iteration=0 
while [ "$iteration" -lt 3 ]
do 
echo "Start of iteration $iteration" 
echo ""
echo "********** START OF NEW TESTCI BENCHMARK JOB **********"
echo "Benchmark Name: LibertyStartupDT Benchmark Variant: 17dev-4way-0-256-qs"
echo "Benchmark Product: pxa6480sr6-20190123_02"
echo ""
# Export benchmark vars
export JDK_OPTIONS="-Xmx256m"

## HW Specific Environment Vars ##
# Export HW vars
bash ./bin/sufp_benchmark.sh 
echo "End of iteration $iteration" 
iteration=$((iteration+1)) 
done

Background About TRS

Once the Jenkins build is done, TestResultSummaryService (TRSS) stores the raw Jenkins build output, which has the output for all benchmark iterations for a build, in MongoDB. Then, TRSS parses the data (i.e. /openjdk-test-tools/TestResultSummaryService/parsers/BenchmarkParser.js) for each iteration by using regex for the string: echo "********** START OF NEW TESTCI BENCHMARK JOB **********", which is printed in Jenkins output for each iteration, and stores the results in another collection in MongoDB. These parsed numbers for various benchmark metrics are used to calculate various important numbers such as average, confidence interval, min, max and median for each build, enabling test-result-summary-client (TRSC) to display performance charts and Perf Compare to compare 2 builds.

Proposed Features for PerfNext:

1. When baseline is not checked, launch the test build using a parent pipeline build that launches child builds for each iteration.
   For example, PerfNext -> Jenkins Parent Pipeline -> Launches Child Parent Pipelines for test build with the sequence: T1, T2, T3
2. When baseline is checked, launch both test and baseline builds using a parent pipeline build that launches child builds for alternate iteration of each baseline and test build in a ping-pong fashion.
   For example, PerfNext -> Jenkins Parent Pipeline -> Launches Child Parent Pipelines with the sequence: T1, B1, T2, B2, T3, B3
3. Output parent pipeline build URL to user

Proposed Features for TRS:

1. Redesign the parser to use the parent pipeline build URL to browse through each child pipeline build and parse data for each iteration

Assigned Contributors

Members from my team, Awsaf (@pinicman) and Sophia (@sophiaxu0424) will be starting to work on the PerfNext and TRS features respectively.

[piyush286]

Jenkins Parent Pipeline should print information in this format. This format would be expected by TRSS for parsing the results.

Snippets from Jenkins Parent Pipeline

Benchmark Name: ILOG_WODM Benchmark Variant: 881-4way-Seg5FastpathRVEJB
ITERATIONS=4
INTERLEAVE=TRUE
Test Build: pxa6480sr6-20181217_01
Baseline Build: pxa6480sr5-20180517_01

Test Build Iteration 1/4
[Pipeline] build (Building PerfNext-Child)
Scheduling project: PerfNext-Child
Starting building: PerfNext-Child #68 
https://jenkinsServer.com/job/PerfNext-Child/68/console

Baseline Build Iteration 1/4
[Pipeline] build (Building PerfNext-Child)
Scheduling project: PerfNext-Child
Starting building: PerfNext-Child #69 
https://jenkinsServer.com/job/PerfNext-Child/69/console

Test Build Iteration 2/4
...

[piyush286]

I'm posting some sketches for reference from our discussions. Apologies for my terrible writing 😄

[piyush286]

[piyush286]

I've posted the outputs and screenshots to show how interleave runs would work.

If you notice the GrandParent build below, it launches parent(s) builds. Each parent would launch children builds. You will notice that Parent build 1 launches child 1 and 3 while Parent build 2 launches child 2 and 4, showing that both baseline and test builds are being interleaved.

Output of PerfNext-GrandParent

Build 53

Started by remote host 9.85.148.226
Running in Durability level: PERFORMANCE_OPTIMIZED
[Pipeline] Start of Pipeline
[Pipeline] lock
Trying to acquire lock on [kermit]
Lock acquired on [kermit]
[Pipeline] {
[Pipeline] parallel
[Pipeline] { (Branch: test)
[Pipeline] { (Branch: baseline)
[Pipeline] build (Building PerfNext-Parent)
Scheduling project: PerfNext-Parent
[Pipeline] build (Building PerfNext-Parent)
Scheduling project: PerfNext-Parent
Starting building: PerfNext-Parent #234
Starting building: PerfNext-Parent #235
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
Lock released on resource [kermit]
[Pipeline] // lock
[Pipeline] End of Pipeline
Finished: SUCCESS

Output of PerfNext-Parent

Build 234

Started by upstream project "PerfNext-GrandParent" build number 53
originally caused by:
 Started by remote host 9.85.148.226
Running in Durability level: PERFORMANCE_OPTIMIZED
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (Build iteration 1)
[Pipeline] echo
Build Iteration 1/2
[Pipeline] build (Building PerfNext-Child)
Scheduling project: PerfNext-Child
Starting building: PerfNext-Child #572
[Pipeline] echo
https://CustomJenkinsServer.company.com/job/PerfNext-Child/572/
[Pipeline] sleep
Sleeping for 15 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build iteration 2)
[Pipeline] echo
Build Iteration 2/2
[Pipeline] build (Building PerfNext-Child)
Scheduling project: PerfNext-Child
Starting building: PerfNext-Child #574
[Pipeline] echo
https://CustomJenkinsServer.company.com/job/PerfNext-Child/574/
[Pipeline] sleep
Sleeping for 15 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS

Output of PerfNext-Parent

Build 235

...
Starting building: PerfNext-Child #573
Starting building: PerfNext-Child #575
...

Output of PerfNext-Child

Build 572

Started by upstream project "PerfNext-Parent" build number 234
originally caused by:
 Started by upstream project "PerfNext-GrandParent" build number 53

Screenshots with Perf Compare