Add Parsers & Perf Graph for AcmeAir

[piyush286]

Problem Description

Parsers need to be added for the following benchmarks so that they can be monitored using the existing tools (Perf Graphs) and tools under development (Tabular View), allowing the summarization and visualization of the perf results.

Benchmarks Parsers to Add

Acmeair: https://github.com/acmeair/acmeair
Octane: https://chromium.github.io/octane/

Steps for Adding a Parser

I'm adding some steps here that I provided in the internal GitHub so that others can refer to this documentation for adding more parsers in future.

Our parsers in Test Result Summary Service (https://github.com/AdoptOpenJDK/openjdk-test-tools/tree/master/TestResultSummaryService/parsers) expect all the results to be in the Jenkins output, so that we don't need to parse results from different files. Hence, we might need to output the results from some log file to the main Jenkins output. If TRSS is monitoring that Jenkins pipeline and it's marked as "Perf" in the build monitoring list (as we discussed in our meeting), then it would parse all the results once the builds are done.

All Jenkins builds should output the following info (Testci string, Benchmark Name, Benchmark Variant, Product) at the start of each benchmark build.

Sample Output:

echo "********** START OF NEW TESTCI BENCHMARK JOB **********"
echo "Benchmark Name: LibertyStartupDT Benchmark Variant: 17dev-4way-0-256-qs"
echo "Benchmark Product: jdk8u181-b13-openj9-0.9.0"

We use the following regexes in TRSS for parsing that info as shown below:

const benchmarkDelimiterRegex = /[\r\n]\*\*\*\*\*\*\*\*\*\* START OF NEW TESTCI BENCHMARK JOB \*\*\*\*\*\*\*\*\*\*[\r\n]/;
const benchmarkNameRegex = /[\r\n]Benchmark Name: (.*) Benchmark Variant: .*[\r\n]/;
const benchmarkVariantRegex = /[\r\n]Benchmark Name: .* Benchmark Variant: (.*)[\r\n]/;
const benchmarkProductRegex = /[\r\n]Benchmark Product: (.*)[\r\n]/;

For startup and footprint, we start an app multiple times (usually 8 times). Before each run, we print "Warm run <Iteration#>" (i.e. outerRegex), which works as an outer regex to get smaller section to parse. Then we use another regex (i.e. regex) to parse the metric value from each smaller section. So we can parse results for each iteration. You can see sample parsed data here (#73 (comment)).

For throughput, we just use one regex (i.e. "Throughput: ") for AcmeAir. AcmeAir can use the same parser for startup, which is already there in the code.

Snippet for Liberty Startup

https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/524d16e3784c17f4af6cee75f9105eb929397792/TestResultSummaryService/parsers/BenchmarkMetric.js#L13-L28
 

Sample Liberty AcmeAir Startup Job:

Warm run 0 //Outer Regex
Footprint (kb)=148944 //Inner Regex
Startup time: 4530 //Inner Regex
...
Warm run 1 //Outer Regex
Footprint (kb)=148120 //Inner Regex
Startup time: 4134 //Inner Regex

...
# Other Iterations

Sample Liberty AcmeAir Throughput Job:

Throughput: 2424.19 //Regex

To Add a Perf Graph

For adding a perf graph widget, you can refer to one of the existing perf graphs such as ODM: https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/test-result-summary-client/src/Dashboard/Widgets/Graph/ODM.jsx

[piyush286]

FYI @jiayaolinn

[piyush286]

@jiayaolinn You'll have to wait for #106 to be fixed before you can add a graph for AcmeAir since Perf Graphs are currently busted because of #73. @sophiaxu0424 will be working on #106 after she finishes her current PR.

[piyush286]

Closing this since all related work to this issue has been completed.