PR: Add manycells and manyoutputs benchmarks on CI and generate artifacts

[goanpeca]

Add CI to run tests and upload artifacts.

Added metadata on JSON

example:

f7b7ee7-manyCells-manyOutputs-5-0.9-100.json

{
 "browsers": {},
 "notebookPackages": [
  "./manyCells",
  "./manyOutputs"
 ],
 "benchmark": {
  "BENCHMARK_OUTPUT": "f7b7ee7-manyCells-manyOutputs-5-0.9-100.csv",
  "BENCHMARK_MAX_N": 100,
  "BENCHMARK_NUMBER_SAMPLES": 21,
  "BENCHMARK_SWITCHES": 10,
  "BENCHMARK_MAX_TIME": 500,
  "TIME_OUT": 300000
 },
 "manyCells": {
  "CELLS_MULTIPLIER": 5,
  "CODE_CELL_RATIO": 0.9
 },
 "manyOutputs": {
  "MANY_OUTPUTS": 100
 },
 "systemInformation": {
  "cpu": {
   "manufacturer": "Intel®",
   "brand": "Xeon® Platinum 8171M",
   "vendor": "GenuineIntel",
   "family": "6",
   "model": "85",
   "stepping": "4",
   "revision": "",
   "voltage": "",
   "speed": "2.60",
   "speedmin": "",
   "speedmax": "",
   "governor": "",
   "cores": 2,
   "physicalCores": 2,
   "processors": 1,
   "socket": "",
   "cache": {
    "l1d": 32768,
    "l1i": 32768,
    "l2": 1048576,
    "l3": 37486592
   }
  },
  "mem": {
   "total": 7292190720,
   "free": 1107718144,
   "used": 6184472576,
   "active": 1356337152,
   "available": 5935853568,
   "buffers": 677113856,
   "cached": 3603525632,
   "slab": 967147520,
   "buffcache": 5247787008,
   "swaptotal": 4294963200,
   "swapused": 536576,
   "swapfree": 4294426624
  },
  "osInfo": {
   "platform": "linux",
   "distro": "Ubuntu",
   "release": "18.04.5 LTS",
   "codename": "Bionic Beaver",
   "kernel": "5.3.0-1034-azure",
   "arch": "x64",
   "hostname": "fv-az65",
   "codepage": "UTF-8",
   "logofile": "ubuntu",
   "serial": "f49a0ad817d846c783a5b12e783fdf6c",
   "build": "",
   "servicepack": "",
   "uefi": false
  }
 }
}

[echarles]

This is awesome @goanpeca . How can we access the uploaded artifacts?

Run actions/upload-artifact@v2
  with:
    name: benchmarks
    path: *.csv
  *.json
  !analysis.vl.json
  !comparison.vl.json
  !package.json
  !tsconfig.json
  
    if-no-files-found: warn
  env:
    JLAB_HOME: /home/runner/work/benchmarks/jupyterlab
    BENCHMARK_NOTEBOOKS: ["./manyCells", "./manyOutputs"]
    BENCHMARK_MAX_N: 100
    BENCHMARK_NUMBER_SAMPLES: 21
    BENCHMARK_SWITCHES: 10
    BENCHMARK_MAX_TIME: 5000
    BENCHMARK_OUTPUT: f7b7ee7-manyCells-manyOutputs-5-0.6-100.csv
    CELLS_MULTIPLIER: 5
    CODE_CELL_RATIO: 0.6
    MANY_OUTPUTS: 100
    pythonLocation: /opt/hostedtoolcache/Python/3.8.5/x64
With the provided path, there will be 2 file(s) uploaded
Total size of all the files uploaded is 4950 bytes
Finished uploading artifact benchmarks. Reported size is 4950 bytes. There were 0 items that failed to upload
Artifact benchmarks has been successfully uploaded!

[echarles]

ok I see them now

[echarles]

@goanpeca A nice addition would be to have the generated notebook (in the data folder) as downloadable artifacts.
This would allow to visually try the generated notebook afterwards in a real jupyterlab.

I am happy to merge we this as-is, but I guess it should not be difficult to enroll that feature before.

[goanpeca]

A nice addition would be to have the generated notebook (in the data folder) as downloadable artifacts.
This would allow to visually try the generated notebook afterwards in a real jupyterlab.

Rigth!, let me add that

[goanpeca]

We need the old being before shadow dom, and the new should be after shadow dom.

🤦 you are right!

[goanpeca]

@echarles is there a way to not repeat this file on the repo src and on this subpackage?

I tried adding an extra include on the root package but they need to be under the same root folder.

I could not bundle this file and copy it on CI maybe?

Just to avoid having two files falling out of sync

[echarles]

I only see one compare.py in this PR. What do I miss here?

[goanpeca]

The ts file on /packages/compare/src/com..ts. Is a 1-1 cooy of yhe same file in /src/

[goanpeca]

I also could not get the vl2png thinguie to work 🤷 , but I guess we can enable that later.

[goanpeca]

Also, the naming convention is:

<commit>-manyCells-manyOutputs-<parameters>.json

Where parameters are the configurable test parameters for manyCells and then for manyOutputs

[echarles]

I also could not get the vl2png thinguie to work 🤷 , but I guess we can enable that later.

later is very good.

[echarles]

Also, the naming convention is: -manyCells-manyOutputs-.json Where parameters are the configurable test parameters for manyCells and then for manyOutputs

Sounds good to me. A new page on the doc site would be great to publish these details for everyone, eg on https://jupyterlab-benchmarks.readthedocs.io/en/latest/benchmarks/ci.html. Maybe add create that page in this PR and paste there some bits

[goanpeca]

Sounds good to me. A new page on the doc site would be great to publish these details for everyone, eg on https://jupyterlab-benchmarks.readthedocs.io/en/latest/benchmarks/ci.html. Maybe add create that page in this PR and paste there some bits

Great idea. Will do!

[echarles]

@goanpeca Thank you so much for this impressive and very useful work. It is working fine and we can iterate with other PR based on opened issues. Merging.