Skip to content

cmd/compile: k8s scalability load test failure (latency regression) with unified IR #54593

Closed

Description

@mknyszek

Kubernetes runs a nightly load test built with Go tip. Recently there was a regression in latency that caused various thresholds to be exceeded.

(See the "gathering measurements step," though that on its own isn't very descriptive.)

The team has narrowed the culprit down to

833367e98a internal/buildcfg: enable unified IR by default

It's unclear at this time what the issue is. We're currently waiting on more metrics to investigate this more thoroughly.

CC @mdempsky

Activity

  1. added
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on Aug 22, 2022
  2. added and removed
    compiler/runtimeIssues related to the Go compiler and/or runtime.
    on Aug 22, 2022
  3. added this to the Go1.20 milestone on Aug 22, 2022
  4. mknyszek commented on Aug 22, 2022

    @mknyszek
    ContributorAuthor

    Assigning to myself for now to gather more information.

  5. self-assigned this
    on Aug 22, 2022
  6. mdempsky commented on Aug 22, 2022

    @mdempsky
    Contributor

    Just to confirm, this is an issue with run-time code being slower? Like it's not a problem with the compiler having slowed down too much?

    One thing that comes to mind is inlining has changed. In particular, the generated IR is somewhat different now, whereas the inlining heuristics are still tuned for the non-unified frontend's IR; and unified IR still has a more primitive heuristic for when to re-export function bodies, so sometimes inlining may fail. And in turn, this might have impacts on escape analysis.

  7. mknyszek commented on Aug 22, 2022

    @mknyszek
    ContributorAuthor

    Nope, it's about code at runtime. Now that I'm looking at some of the graphs, it actually looks more like occasional stalls than a uniform slowdown, though I think there is some smaller uniform impact on tail latencies as well in between those stalls (unclear if they're related yet).

    Here's what the API call latency looks like over time (note the threshold for the first graph is 1 second):

    image

    And here's some GC-related statistics (note the breaks in the graphs that line up with the spikes):

    image

  8. mdempsky commented on Aug 30, 2022

    @mdempsky
    Contributor

    Kubernetes reports this is fixed since 6a801d3.

  9. Repository owner moved this from In Progress to Done in Go Compiler / Runtime / Commandon Aug 30, 2022
  10. locked and limited conversation to collaborators on Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions