Refactor BenchmarkReporter

[trungleduc]

References

Follow-up on benchmark in the CI

Code changes

• Refactor BenchmarkReporter to make markdown table and Vega-Lite graph customizable.
• Add tests for BenchmarkReporter

User-facing changes

N/A

Backwards-incompatible changes

N/A

[jupyterlab-probot]

Thanks for making a pull request to jupyterlab!
To try out this branch on binder, follow this link:

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

Results table

Test file	large_code_notebook	large_md_notebook
open
chromium
actual	6693 <- [6959 - 7175 - 7475] -> 8082	3326 <- [3392 - 3483 - 3678] -> 3771
expected	6938 <- [7584 - 7618 - 7760] -> 8204	3461 <- [3524 - 3534 - 3590] -> 4136
switch-from
chromium
actual	393 <- [413 - 423 - 442] -> 464	278 <- [304 - 311 - 328] -> 362
expected	464 <- [480 - 489 - 499] -> 789	310 <- [338 - 344 - 352] -> 370
switch-to
chromium
actual	1549 <- [1657 - 1687 - 1718] -> 1788	1464 <- [1483 - 1501 - 1538] -> 1603
expected	1706 <- [1730 - 1764 - 1814] -> 2015	1541 <- [1549 - 1562 - 1574] -> 1608
close
chromium
actual	5355 <- [8411 - 12762 - 15520] -> 16102	3530 <- [3716 - 3887 - 4067] -> 4413
expected	5176 <- [5770 - 10533 - 14493] -> 18824	3325 <- [3691 - 3890 - 3939] -> 4073

❗ Test metadata have changed

--- /dev/fd/63	2021-09-17 08:42:55.074206707 +0000
+++ /dev/fd/62	2021-09-17 08:42:55.074206707 +0000
@@ -8,33 +8,33 @@
   },
   "systemInformation": {
     "cpu": {
-      "brand": "Xeon® E5-2673 v4",
+      "brand": "Xeon® Platinum 8171M",
       "cache": {
         "l1d": 65536,
         "l1i": 65536,
-        "l2": 524288,
-        "l3": 52428800
+        "l2": 2097152,
+        "l3": 36700160
       },
       "cores": 2,
       "family": "6",
-      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear",
+      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear",
       "governor": "",
       "manufacturer": "Intel®",
-      "model": "79",
+      "model": "85",
       "physicalCores": 2,
       "processors": 1,
       "revision": "",
       "socket": "",
-      "speed": 2.3,
+      "speed": 2.6,
       "speedMax": null,
       "speedMin": null,
-      "stepping": "1",
+      "stepping": "4",
       "vendor": "GenuineIntel",
       "virtualization": false,
       "voltage": ""
     },
     "mem": {
-      "total": 7291699200
+      "total": 7291695104
     },
     "osInfo": {
       "arch": "x64",
@@ -42,11 +42,11 @@
       "codename": "Focal Fossa",
       "codepage": "UTF-8",
       "distro": "Ubuntu",
-      "kernel": "5.8.0-1041-azure",
+      "kernel": "5.8.0-1040-azure",
       "logofile": "ubuntu",
       "platform": "linux",
       "release": "20.04.3 LTS",
-      "serial": "faac5b9e00cf4ae3b2d2c013e98d933a",
+      "serial": "0f5fd491ff264b5f9d56e03599b4fae0",
       "servicepack": "",
       "uefi": false
     }

[fcollonval]

Thanks a lot for doing this @trungleduc

My main comments are to change a bit the signature of the new arguments to be more generic and more explicit.

We should also add some documentation about that reporter in galata README. Would you mind adding please?

[fcollonval]

Let's clarify that the output is expected to be a VegaList configuration.

Suggested change

	graphConfigFactory?: (
	allData: Array<IReportRecord>,
	comparison: 'snapshot' | 'project'
	) => Record<string, any>;
	vegaLiteConfigFactory?: (
	allData: Array<IReportRecord>,
	comparison: 'snapshot' | 'project'
	) => JSONObject;

[trungleduc]

Thanks @fcollonval for reviewing it.
Comments resolved in last commit.

[fcollonval]

Here are some additional suggestions.

[fcollonval]

Thanks @trungleduc