Skip to content

runtime: average stack size is not a fit for every application #61052

Open
@cdvr1993

Description

@cdvr1993

In go1.19 was introduced that stack are created basen on the historical average size. Even though this has improved the landscape for our applications, we still experience issues with some of them.

To give one real example. Let's introduce a 15k cores application. Originally the profiles were showing close to 10% cpu being consumed by stack growth. If we translate this to cores that's ~1.5k cores.

Screenshot 2023-06-28 at 11 50 07 AM

The stack usage is very low (we never exceed 16M):

Screenshot 2023-06-28 at 11 52 19 AM

After adding to our metrics to publish the runtime/metrics that exposes the initial stack size we could observe that 99% of the time the stack size is set to 2KB.

By using go:linkname we were able to expose the initial stack size global variable. So what we did was to enable "gcshrinkstackoff" and disable "adaptivestackstart", to not shrink stacks and to avoid the runtime overriding our value.

Screenshot 2023-06-28 at 12 38 26 PM
Screenshot 2023-06-28 at 12 38 14 PM
Screenshot 2023-06-28 at 12 39 18 PM

Finally we ran several experiments injecting different stack sizes:

4KB
Screenshot 2023-06-28 at 11 58 13 AM

8KB
Screenshot 2023-06-28 at 11 58 39 AM

16KB
Screenshot 2023-06-28 at 11 59 05 AM

We decided to stop at 16KB because the gains were approaching 0, but as you can see we were able to reduce copystack by 7%.

Memory went up, but it is still reasonable. From 16MB to 50MB.
Screenshot 2023-06-28 at 12 02 57 PM

With all that said, would it be possible for the Go runtime to expose:

  • A safe way to inject our own stack size (even if it is static).
  • A histogram with all the stack sizes it has seen, this would helps decide what size would gives us the best results without using too much memory.

Or do you have any idea on how to implement this more safely (not requiring private linking).

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.compiler/runtimeIssues related to the Go compiler and/or runtime.

    Type

    No type

    Projects

    Status

    Todo

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions