Create/adapt dashboards covering major Coder architectural components

[dannykopping]

• Controlplane
  • Number of replicas
  • API requests per route per replica
  • CPU per replica (requests, limits, usage)
  • Memory per replica (requests, limits, heap, RSS, gc pacer)
  • Pod restarts
  • Pubsub latency
  • Active users vs license seats
• Workspaces
  • Pod restarts
  • Pod OOMs
  • Pods with fullest disks (requires https://github.com/kubernetes-sigs/metrics-server/ to be installed)
  • Pods with highest CPU usage
  • Pods with highest memory usage
• Workspaces Detail (inspiration)
  • Logs for workspace
  • Template name & version
  • Owner
  • Build status
• Workspace proxies
  • Status
• Provisioner
  • Number of built-in provisioners
  • Number of external provisioners
  • Build successes vs failures
  • Build successes vs failures (per template)
  • Builds in progress
• Networking
  • Websocket
    • Current connections
    • Traffic by workspaces
    • Traffic by coderd replica
  • Agent
    • Current connections
    • Traffic by workspaces
    • Traffic by coderd replica
• Health checks (stretch)

[dannykopping]

Make sure we include workspace OOMs in one of the dashboards: https://github.com/coder/customers/issues/584#issuecomment-2093566479

[bpmct]

@dannykopping Can we announce the beta without these remaining dashboards and move these over to separate issues in coder/observability? Asking because we have a lot to cover over in June (notifications, etc) and customer feedback can influence our priorities for the observability helm chart.

[dannykopping]

I would argue so, yeah 👍