runtime: better adaptive stack sizing

[geekswaroop]

Hello Go maintainers,

Related to #61052 #18138, #7514, but bringing more data.
This was discussed in #57175 recently, raising an issue for a future reference.

We have recently built capabilities to measure the cost of stack growth across our fleet. This cost is significant (3.9%, core weighted) vs a traditionally costly thing like GC, which costs us 7.3%.

We would like to solve this, we have some ideas, but we hope that by providing the data we can land a solution that is acceptable witihin OSS. We’re happy to contribute once you advise us on the best approach.

Fleet wide costs of runtime.newstack:

Scope	Cores of the fleet	Copystack cost	More than 3%	More than 2%	More than 1%	Average
Top10	33%	30%	8/10	10/10	10/10	4.3%
Top20	48%	42%	12/20	14/20	18/20	3.7%
Top100	82%	80%	62/100	73/100	88/100	3.7%
~600	100%	100%	334/600	406/600	509/600	3.9%

Top 13 services goroutine stack distribution:

Service	Mean	P50	P75	P90	P99	Max	Starting stack size
1	14.2 KB	12.1 KB	21.1 KB	22.2 KB	23.3 KB	25.1 KB	4kb
2	12.5 KB	10.6 KB	20.1 KB	25.9 KB	31.4 KB	31.6 KB	4kb
3	15.5 KB	13 KB	23.2 KB	24.2 KB	24.8 KB	24.9 KB	4kb
4	16.0 KB	12 KB	26.9 KB	30.7 KB	37.8 KB	38.0 KB	4kb
5	19.0 KB	19.3 KB	23.6 KB	32.3 KB	42.9 KB	43.1 KB	4kb
6	17.0 KB	11.9 KB	21.6 KB	36.1 KB	72.5 KB	73.4 KB	4kb
7	13.1 KB	12 KB	16.6 KB	22.1 KB	56.5 KB	56.9 KB
8	10.1 KB	7.1 KB	12.0 KB	21.1 KB	22.9 KB	23.1 KB
9	6.9 KB	6.1 KB	11.7 KB	12.4 KB	13.1 KB	13.6 KB
10	5.0 KB	3.5 KB	5.4 KB	10.5 KB	22.5 KB	22.6 KB	2kb
11	24.9 KB	26.6 KB	30.7 KB	31.3 KB	31.4 KB	32.2 KB	4kb
12	6.4 KB	2.7 KB	7.0 KB	21.2 KB	21.4 KB	21.4 KB	2kb
13	12.9 KB	9.9 KB	22.2 KB	22.7 KB	24.5 KB	38.4 KB	4kb

Top13 services goroutine stack categorization:

Service	≥ 4KB	≥ 8KB	≥ 16KB	≥ 24KB	≥ 32KB	≥ 48KB	≥ 64KB
1	99.5%	73.9%	47.1%	0.5%
2	99.75%	61.4%	27.2%	15.2%
3	100%	73.6%	44.0%	13.6%
4	98%	67.4%	39.9%	32.4%	4.7%
5	97.2%	82.0%	66.5%	24.9%	10.5%
6	98.4%	74.8%	42.6%	22.1%	10.9%	3.5%	1.2%
7	98.4%	65.9%	34.1%	4.1%	3.3%	3.3%
8	99.1%	49.6%	13.3%
9	61.1%	34.3%
10	46.6%	12.7%	2.5%
11	100%	91.3%	84.5%	80.7%	0.6%
12	42%	24.0%	10.0%
13	90%	75.5%	36.2%	1.9%	0.2%

For now, we have a per-service knob that overwrites those values and have been rolling overrides to either 16kb or 32kb for the top services. Those changes worked as expected, we see dramatic CPU drop and corresponding stack memory increase - in our case it’s usually <200mb of memory and it’s acceptable for those services as their heap sizes are much larger.

For a long term plan we’re considering several ideas, we might be trying some of them internally.

Ideas	Comment
New stats and that’s it (number of growth/shrink, histogram?)
Emit copystack cost, similar to the GC cost.
Min-size knob	Dumb ideas 😞
Override min 4k for everyone?
“High memory mode” similar to GOMEMLIMIT?
Round up to n+1 size
P75 instead of mean would be good enough	Reduces growth ops by ~50% in our fleet.
Use new metrics: growth vs goroutine start to adjust dynamically	Complicated (?)
Start higher: allow for faster shrinking	“Optimistic presizing”
Start higher: shrink for “stable” goroutines
Limit scope: Exclude long running goroutines	“Excluding outliers”
Limit scope: Only include goroutines that had stack growth in the last N cycles
Limit scope: Only include goroutines that started (or died?) in the last N cycles

[gabyhelp]

Related Issues

• runtime: average stack size is not a fit for every application #61052
• runtime: dynamically adjust initial stack size according to workload #31558 (closed)
• runtime: new goroutines can spend excessive time in morestack #18138
• runtime: copy and expand stacks instead of splitting them #4910 (closed)
• cmd/pprof, runtime/pprof: add stack size profiler #20010 (closed)
• proposal: runtime/pprof: add goroutine stack memory usage profile #66566
• runtime: set initial stack size to 4 kB #7514 (closed)

Related Discussions

• Go Stack Design Proposal
• Go 1.19 average goroutine stack
• Go 1.19 stack size optimization

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dr2chase]

@randall77

[randall77]

Service	Mean	P50	P75	P90	P99	Max	Starting stack size
1	14.2 KB	12.1 KB	21.1 KB	22.2 KB	23.3 KB	25.1 KB	4kb

That doesn't look right. We have adaptive stack starting sizes now. I would think that the code here would decide on a 16KB starting stack if the mean stack size is 14.2KB.

How are you measuring mean stack size? Does that include all goroutines, or only the main servicing goroutines?

[cdvr1993]

Service
Mean
P50
P75
P90
P99
Max
Starting stack size

1
14.2 KB
12.1 KB
21.1 KB
22.2 KB
23.3 KB
25.1 KB
4kb

That doesn't look right. We have adaptive stack starting sizes now. I would think that the code here would decide on a 16KB starting stack if the mean stack size is 14.2KB.

How are you measuring mean stack size? Does that include all goroutines, or only the main servicing goroutines?

We are getting this numbers from profiles. So this is only analyzing goroutines that actually needed expansion (any goroutine that was lower than starting stack size won't show in the profile), so yeah this is only mean of goroutines exceeeding current mean.

[randall77]

Ok, so that makes me think that your goroutine stack size distribution is bimodal. It looks like the average size here is below 4KB (because that's the size the runtime chose for a starting stack), but for the goroutines whose stack exceeds 4KB, they are quite a bit larger.

For now, we have a per-service knob that overwrites those values and have been rolling overrides to either 16kb or 32kb for the top services. Those changes worked as expected, we see dramatic CPU drop and corresponding stack memory increase - in our case it’s usually <200mb of memory and it’s acceptable for those services as their heap sizes are much larger.

Does the 200MB space increase include the space for the small stacks getting larger, or does it just count the space from the big stacks getting larger?
How large is the total heap?

Possibly one way to go here is to increase the starting stack size to, say, the 75th percentile of existing stack sizes, or even the 90th percentile, if the total size for all stacks would remain under, say, 1% of the total heap size. By bounding to the total heap size we ensure that the extra stack space we waste is not significant overall in the application.

And we could always use more optimization of the stack copying itself. I don't think anyone has looked at it in quite a while.

[cdvr1993]

One idea on the top of my head was... We already track GC cpu time and Scavenger cpu time... Could we do something similar for stack growth? Like if current cpu time is <1% do nothing, but if it is >1% instead of using mean use p75... If it is not enough do p90 and so on. That way we are only impacting applications that see a lot of stack growth (I know we put limitation to things like the scavenger, but not sure how much effort is to do this).

It might be just easier to do a simple higher percentile though.

[thepudds]

On Thursday in #57175, one of the Uber folks said for their data:

The problem with the global average seems to be primarily long-dormant goroutines skewing the average.

Based on that, I have a local change that attempts to change the average stack size calculation to ignore goroutines that have been inactive for ~one GC cycle.

That seems to reduce CPU usage in a synthetic benchmark with many small inactive goroutines with other goroutines that grow the stack.

I need to do a little bit more sanity checking, but I will probably send a CL for discussion (maybe either later tonight or maybe tomorrow).

It’s very much exploratory and intended to help move the conversation forward.

Maybe another solution is better, or maybe something like that makes sense in addition to some other change, like looking at percentiles.

[rabbbit]

@randall77 yeah bimodal makes sense.

After your message (sorry, we should have given those details from the start), I looked at the "service 1" that you quoted. Randomly samping a single instance I see 3678 goroutines.

There are:

• 480 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer) with tiny stack
• • I don't grok grpc enough yet to know what this is, but it seems relatively silly - most of these goroutines are alive since the start of the service
• around 440 of our internal kafka publisher library that seems to spin up 10 worker goroutines per destination topic (new discovery right now). The stack is very shallow there.
• 702 goroutines in internal/poll.runtime_pollWait
• 216 goroutines in base/chrono/periodic.(*defaultRunner).execAndWait - basically a timer, I would expect to be various internal libraries "refreshing their caches."
• around 200 in google.golang.org/grpc/internal/transport.* with also fairly shallow stack
• 312 in tchannel-go.(*Connection).writeFrames - like above.

Does the 200MB space increase include the space for the small stacks getting larger, or does it just count the space from the big stacks getting larger?
How large is the total heap?

This would largely depend on the application - the one quoted above (with the biggest growth, where we override the dynamic growth with 32kb) had a memory limit of 48gb.
The 200MB was just a growth in stack size, that service happend to have around 7000-8000 goroutines per instance.

@thepudds if the AI is not lying to me, yeah, that should work

Raw data, click to unlock

Here are the goroutines grouped by state and duration:
Count State
1022 select (no duration — active)
827 select, 194 minutes
414 select, 14 minutes
309 IO wait (no duration)
239 select, 15 minutes
133 IO wait, 14 minutes
93 select, 3 minutes
76 IO wait, 15 minutes
71 select, 1 minutes
55 chan receive (no duration)
45 IO wait, 3 minutes
38 select, 2 minutes
37 chan receive, 194 minutes
33 IO wait, 1 minutes
33 select, 6 minutes
31 select, 4 minutes
26 IO wait, 2 minutes
26 IO wait, 6 minutes
16 select, 12 minutes
15 runnable
14 IO wait, 194 minutes
14 select, 8 minutes
10 IO wait, 8 minutes

Key observations:

• 827 goroutines have been stuck in select for 194 minutes (the full process uptime) — these are long-lived background loops (config watchers, metric reporters, etc.)
• ~650 goroutines stuck at 14-15 minutes — likely a recent burst of activity (requests or connections) that haven't cleaned up
  1,022 select and 309 IO wait without durations are currently active/recently entered their wait state
• The bulk of goroutines are in select (2,696) or IO wait (693), with a small number in chan receive (92)

(I don't actually know what 14 minutes in goroutine 927013079 [IO wait, 14 minutes] is - time since last activity?)

It seems both:

• only count active goroutines, or
• only count goroutines that died recently (to ignore the background ones, which really shouldn't be affecting the starting ones size?)

are likely to be a good heuristic.

This also means explains why grafana/mimir#7131 worked well. We have a few entry points like the grpc one (a "worker" framework that spawns goroutines, a "safego" framework that wraps goroutines to recover panics (😞 )). If we added goroutine pools there, we could feasibly minimize the impact as well.