Create Thundering Herd Autostart load generator

[ethanndickson]

From the plan:

Goal

• To observe how Coder handles a 9am autostart rush, at scale.

User target	coderd replicas	Provisioners (w & w/o queuing)	Concurrent builds
1k	3	60 & 300	300
2k	3	120 & 600	600
3k	4	240 & 900	900
10k	10	1200 & 3000	3000

Infrastructure:

• One Kubernetes cluster in one region.

Implementation:

• Tools:
  • scaletest create-workspaces
    • Requires the ability to set the autostart time of those created workspaces.

Scenarios:

1. Without builds queued.
   • Where the number of provisioners is equal to the number of concurrent builds
2. With builds queued.
   • Where the number of provisioners does exceed the number of concurrent builds

Run sequence:

1. Create users & workspaces.
2. Stop workspaces.
3. Configure a single autostart time in the near future for all workspaces.
4. Wait for all workspaces to have been started.

Key Results:

• How quickly the jobs get created.
• How quickly they get queued.
• How long it takes for all workspaces to be built.
• How long it takes for a workspace to be built (p50/p95/p99).
• Any discrepancies in these results between scenarios.

Other results:

• Coder deployment + database resource utilization (incl. pprofs, Grafana Pyroscope).
• Workspace creation success rate.