cmd/compile: k8s scalability load test failure (latency regression) with unified IR

[mknyszek]

Kubernetes runs a nightly load test built with Go tip. Recently there was a regression in latency that caused various thresholds to be exceeded.

(See the "gathering measurements step," though that on its own isn't very descriptive.)

The team has narrowed the culprit down to

833367e98a internal/buildcfg: enable unified IR by default

It's unclear at this time what the issue is. We're currently waiting on more metrics to investigate this more thoroughly.

CC @mdempsky

[mknyszek]

Assigning to myself for now to gather more information.

[mdempsky]

Just to confirm, this is an issue with run-time code being slower? Like it's not a problem with the compiler having slowed down too much?

One thing that comes to mind is inlining has changed. In particular, the generated IR is somewhat different now, whereas the inlining heuristics are still tuned for the non-unified frontend's IR; and unified IR still has a more primitive heuristic for when to re-export function bodies, so sometimes inlining may fail. And in turn, this might have impacts on escape analysis.

[mknyszek]

Nope, it's about code at runtime. Now that I'm looking at some of the graphs, it actually looks more like occasional stalls than a uniform slowdown, though I think there is some smaller uniform impact on tail latencies as well in between those stalls (unclear if they're related yet).

Here's what the API call latency looks like over time (note the threshold for the first graph is 1 second):

And here's some GC-related statistics (note the breaks in the graphs that line up with the spikes):