feat: make agent stats' cardinality configurable

[dannykopping]

Implements #12221

When stats from agents are collected, they are aggregated by 4 dimensions: agent_name, template_name, workspace_name, and username. This can result in some very high cardinality metrics being scraped by Prometheus in large environments.

This PR adds the ability to tune which labels are include in this aggregation, therefore reducing the cardinality.

For example:

agent_sessions_total{agent_name="main",magic_type="ssh",pty="yes",template_name="docker",username="danny",workspace_name="test1"} 6
agent_sessions_total{agent_name="main",magic_type="ssh",pty="yes",template_name="docker",username="danny",workspace_name="test2"} 5

With hundreds of active workspaces, each having a unique name, the cardinality may be unacceptable. In this case the operator may choose to configure CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY=username to rather aggregate by user, summing all the metrics' values and only producing a single metric series:

agent_sessions_total{magic_type="ssh",pty="yes",username="danny"} 11

Multiple labels can be provided, e.g. CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY=username,agent_name

agent_sessions_total{agent_name="main",magic_type="ssh",pty="yes",username="danny"} 11

The current behaviour remains the default; if no value is passed to CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY there will be no change.

Note to reviewer: I made a sweeping refactor to define the label names in a single location instead of duplicating them, which may make the PR larger than it seems.

[mafredri]

Noticed some minor nits, but in general look good, nice work!

[mafredri]

Unfortunately our formatter doesn't handle merging import groups and leaves things in a messy state (depending on what program injected them). 😔

If you notice these, please feel free to fix, but the standard is we try our best but sometimes these slip through, so don't worry too much.

[mafredri]

I think using map[string]bool would be preferable/clearer here, also simplifies the map lookup later. Typically I'd use either bool or if we're looking for space savings, I'd use the zero struct (map[string]struct{}).

[mafredri]

LGTM! I know tests aren't passing now but looks unrelated so I don't need to re-review, nice work!

[mtojek]

Nice!

I know tests aren't passing now but looks unrelated so I don't need to re-review, nice work!

Agree, feel free to t.Skip them for now.