Change https://go.dev/cl/416417 mentions this issue: runtime: overestimate the amount of allocated memory in heapLive

I've updated the top post, but there's only one regression left to get to the bottom of before release. I expect to resolve that this week and I'm on top of it.

I may have narrowed down the cause of the tile38 memory regression on tip (and the original culprit I suspect seems innocent now; reverting on tip changes nothing).

My guess is that it's related to the fact that the GC pacer math now, in some places, uses the actual trigger point over the precomputed GOGC-based trigger. This is technically "more" correct because that math (like the cons/mark calculation) really does care more about how much was actually allocated than it does when we intended to trigger. It also really shouldn't make much of a difference in most cases.

However, where it does make a difference is in the PI controller used to help smooth the cons/mark measurements a bit. Because we're using the actual trigger point now, there are a bunch of recalculations of cons/mark that are skipped because, as it turns out, during the initialization phase of the tile38 server, the bytes of allocation between the trigger point and when the GC is done is zero. We can't do anything with that because it can cause divide-by-zero issues, and it's also just not very informative.

But these are cases where the 1.18 pacer takes that information (because it's using a lower precomputed trigger over the actual trigger point) and decides to flail wildly for a few GCs. This flailing now ends up in the PI controller's history and by the time we actually get to the main phase of the server (the part we actually measure) the PI controller ends up insisting that the cons/mark ratio is higher than the samples its getting (and would take more GCs than the benchmark currently executes to settle). The actual measured cons/mark in this phase between Go 1.18 and Go 1.19 is roughly the same across the GC cycles. Go 1.19's pacer doesn't flail like this, so it ends up underestimating the cons/mark a bit.

This difference in cons/mark causes Go 1.19's pacer to give the GC a lot less runway, and sometimes the underestimate swings low enough that it's not enough runway, incurring more assists. More GC cycles in the benchmark would smooth this out, but the benchmark already runs for a long time as it is.

Also as result of this fix, Go 1.19's pacer tends to be a little more on-target when it comes to the heap goal, which is why it ends up using slightly more memory.

The end result is: more memory used, roughly equal performance (throughput and latency). we don't get the benefits of fewer GC cycles on average because of (AFAICT) the additional assists that trigger due to the occasional insufficient runway.

One thing that I'm kind of starting to dislike in all this is the PI controller. I'm regretting the decision to use that as a smoothing function now. Yes, it's the only thing that has a chance at really truly finding the steady-state, but it's pretty swingy and I don't think we can afford to damp it more.

I've been experimenting with a much simpler smoothing function: moving average cons/mark over the last 2 cycles. it appears to work OK, is far more stable, and on at least two of the tile38 benchmarks turns down the memory use regression a bit. It also will prevent these kinds of surprises in the future.

This seems to work OK across the board in terms of mitigating the regression.

As an alternative, I am looking into switching back to the precomputed trigger to see if it also resolves the issues.

Good news, looks like using the precomputed trigger resolves the remaining issues. That's probably safer for Go 1.19, so I'm going to advocate for landing that. For Go 1.20, I think I'd like to go back to using the actual trigger and switching to a simpler smoothing function, like the 2-GC moving average I described above.

Change https://go.dev/cl/417557 mentions this issue: runtime: revert to using the precomputed trigger for pacer calculations