Execute tests with the existing load generator tests

[ibetitsmike]

From the plan:

Connection Scaletesting

Goal:

• To observe how Coder reacts workspace connections at scale.

User target	Workspaces	Regions	coderd replicas	Provisioners	Workspace Proxies
1k	600	3	3	60	2
2k	1200	3	3	120	4
3k	1800	3	4	180	6
10k	6000	3	10	600	20

Infrastructure:

• Kubernetes cluster per region, each cluster has:
  • 1/3 of total workspaces.
  • 1/3 of total provisioners.
  • 1/2 of the total workspace proxies in secondary regions (those w/o the coder deployment).

Implementation:

• Tools: scaletest create-workspaces + scaletest workspace-traffic
• 1024 bytes @ 100Hz .
  • Previously established average traffic for a connection.
• Build concurrency equal to provisioners, to minimize build load.

Scenarios:

1. ReconnectingPTY.
2. SSH.
3. SSH (direct connections disabled).
4. Workspace apps.
   • With clients distributed evenly amongst regions, to test workspace proxies.

Run sequence:

1. Create users & staggered start workspaces.
2. Establish baseline over 10 minutes (per current setup).
3. Start workspace-traffic (with the corresponding connection type set) to all workspaces.
4. Run for 15 minutes (per current setup).

Key results:

• Connection throughput.
  • Anything less than 90% of the target throughput, as we scale past 3k users, is going to be unacceptable.
• Client disconnect rate.
  • <1% of clients disconnecting is likely acceptable .
• Any discrepancies in these results between scenarios.

Other results:

• Coder deployment + database resource utilisation (incl. pprofs, Grafana Pyroscope).
• Workspace creation success rate.
• API response times.