rpc: add error_code label to psrpc error_total metric

[paulwe]

Why

livekit_psrpc_error_total counted failed RPCs with only role, kind, service, method. When it climbed there was no way to tell how the RPCs failed — a burst of unavailable (load/affinity), deadline_exceeded (timeouts) and permission_denied (caller misconfiguration) were indistinguishable, so every spike needed a log dive to triage. psrpc already carries a bounded error taxonomy (psrpc.ErrorCode, ~23 constants) that was being thrown away at the metrics boundary.

What

• error_code added as a trailing label on error_total only; the other five collectors are untouched.
• Values come from a small errorCodeLabel helper wrapping psrpc's existing GetErrorCode, so the label domain is exactly psrpc's ErrorCode constants — err.Error() is never used as a label value, keeping cardinality bounded. psrpc.OK (the empty string) and unresolvable errors both collapse to unknown.
• Label slices now use slices.Concat instead of append. This is a correctness prerequisite, not cleanup: bytesLabels := append(labels, …) only reallocates today because maps.Keys happens to return len == cap. Adding a second append(labels, …) for errorLabels would make both slices write index len(labels) of the same backing array whenever labels had spare capacity, silently registering one collector with the other's label name.

Known limitation

Multi requests always report error_code="unknown". psrpc's middleware.MetricsObserver.OnMultiRequest has no error parameter and multiRPCMetricsInterceptor.Recv drops the error, so the code isn't available. These rows stay separable via the existing kind="multirpc". Surfacing real codes would need an err param on that psrpc interface — breaking, though the only implementors anywhere are the two observers in this same file.

Bare context.Canceled / context.DeadlineExceeded also land in unknown, since psrpc doesn't special-case them. Left alone deliberately — if kind="stream" turns out to be dominated by unknown in practice, that's a two-case pre-check to add later.

Rollout

Adding a label changes the series identity of livekit_psrpc_error_total. Aggregating queries are unaffected — the production alert rules use sum(rate(livekit_psrpc_error_total{...})). Expect a one-scrape discontinuity as old series are replaced by new ones (normal counter-reset handling in rate()). Only queries matching the full label set exactly would need updating.

Test plan

• go build ./..., go vet ./rpc/, full suite green (30 packages).
• New rpc/metrics_test.go drives all four error paths and gathers from the default registry. Deliberately avoids prometheus/testutil — its kylelemons/godebug dep is absent from go.sum and importing it would force a go.mod change.
• Mutation-checked the new test: broke errorCodeLabel to return "", confirmed the test fails, reverted.
• Verified the curried-label path end-to-end against cloud-protocol's real config (nats_server curry label), since that's where the WithLabelValues arity risk lives: error_code=deadline_exceeded landed alongside nats_server=nats-a.
• Downstream go build in cloud-protocol / cloud-io / cloud-egress / egress. cloud-io, cloud-egress and egress fail, but on a stale backend-common/observability/gatewayobsimpl generated reporter and an egress embed pattern — confirmed identical failures with this change stashed. None touch rpc metrics.

[changeset-bot]

🦋 Changeset detected

Latest commit: 80890e9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR