Create load generator for Prebuilds

[ethanndickson]

From the plan:

Goal:

• To observe how Coder handles prebuild reconciliation at scale

User target	Regions	Templates	Prebuilds per template	Presets per template	Provisioners	Concurrent Reconciliations
1k	3	6	10	5	60	60
2k	3	12	10	5	120	120
3k	3	18	10	5	180	180
10k	3	60	10	5	600	600

Infrastructure:

• Kubernetes cluster per region, each cluster has:
  • 1/3 of total prebuilds
  • 1/3 of total provisioners

Implementation:

• Tools:
  • coder scaletest command for bulk creating templates with prebuilds
• A baseline template
  • With a schedule to create one prebuild instance, and then shortly after, destroy it.
• Multiple templates, each:
  • With a schedule to create all prebuilds instances at the same time
  • With a schedule to delete all prebuild instances at the same time
  • With presets to spread the resulting prebuilds across regions

Run Sequence:

1. Create a user
2. Determine a start time and an expiration time for all prebuilds, in the near future.
3. Create all templates
4. Await baseline creation time & baseline expiry time.
5. Await mass-creation time
6. Await mass-expiry time

Key Results

• How quickly the jobs get created.
• How quickly they get queued.
• How long it takes for all workspaces to be built.
• How long it takes for a workspace to be built, relative to the baseline.

Other Results:

• Coder deployment + database resource utilisation (incl. pprofs, Grafana Pyroscope)