Performance regression testing

[chrisglein]

Goals and plan document (internal link)

• Run baseline perf apps against performance infrastructure #9416
• Create perf pipeline #9417
  • Create performance yml file #10114
    Performance pipeline sets up a Azure hosted VM with RNW secrets/variables, gets the latest published version of RNW and builds it. (3 days)
    • Copy compliance.yml
    • Rename to performance.yml
    • Remove compliance specific jobs
  • Add HelloWorld scenario job #10115
    Pipeline now builds, deploys, and launches the apps on the VM (2 days)
    • Build app
    • Deploy app
    • Run scenario (launch)
  • App perf measurement to scenario job #10116
    Pipeline now gathers perf traces of the app and store the results in a CSV file (6 days)
    • Run machine quieting scripts - 1 day
    • Write perf profiles - 1 day if copying existing… unknown if we actually have to write our own
      • Copy WinUI profiles
      • Copy the profile in vnext/Scripts/Tracing (Hermes specific?)
    • Extract data using perf tool + profiles - 2 days
    • Store data to CSV - 3 days
      • Figure out where CSV "lives" (temp? artifacts?) - 1 day
      • Create CSV - 1 day
      • Append data to CSV - 1 day
  • Create template for gathering traces on any scenario #10117
    Simplifies app launch + perf gathering yml to template asking for app name (5 days)
    • Learn more yml - 1 day
    • Add run-perf-scenario.yml - 2 days
    • Rewrite HelloWorld job to use run-perf-scenario.yml - 2 days
• Add MongoDB database #9418
  Perf results are now stored in a database (Unknown - 8 days?)
  • Learn MongoDB - 2 days
  • Set up database machine - 2 days
  • Create database using dashboard guidelines- 2 days
  • Automate perf data storage from CSV to database - 2 days
• Add perf dashboard to github #9419
  Can now track day to day perf changes in graphs/dashboard (Unknown - 5 days?)
  • Figure out what metrics we want to display
  • Learn dashboard schema
• Expand test suite - test priority matrix:
  • Build baseline perf app(s) #4872

Scenario \ Platform	RNW	WASDK (WebView2)	XAML	Electron	Flutter
Baseline	0	1	1	2	3
Real-World Example	0	1	1	2	3
Multiple Instances	1	2	2	3	4
Input	1	2	2	3	4
Collections	1	2	2	3	4
File System	2	3	3	4	5
Media	2	3	3	4	5

[chrisglein]

Notes from planning poker:

1. need to build some number of test apps (growing this should be costed separately). Start with existing apps (hello world, gallery, anything else is bonus). More apps = more comparison, but that should come after we have a pipeline that works with the most basic single app.
2. need to integrate into whatever system the perf team has (there will be some waiting for their output and evaluating whether it meets our needs). Assume what they have is new and not initially broadly tested (growing from a pile 'o scripts to a something more robust)
3. then enable that in our CI
4. how do we store and view long-term results for comparative analysis? (does the perf team have tech/practices here?)
   Regression testing = the end result. We need to cost the steps along the way.
   A plan that involves iteration here is what we need. This will expand over time from something very modest.