runtime: GC pacing exhibits strange behavior with a low GOGC

[mknyszek]

As of Go 1.14, GC pacing changed a bit to alleviate issues with an increased allocation rate, related to #35112.

Unfortunately, the pacing change that was made was made in error. Currently, the code for capping the trigger ratio in gcSetTriggerRatio looks like:

    const minTriggerRatio = 0.6                                                                                                                                                                             
                                                                                                                                                                                                            
    // Set the trigger ratio, capped to reasonable bounds.                                                                                                                                                  
    if triggerRatio < minTriggerRatio {                                                                                                                                                                     
        // This can happen if the mutator is allocating very                                                                                                                                                
        // quickly or the GC is scanning very slowly.                                                                                                                                                       
        triggerRatio = minTriggerRatio                                                                                                                                                                      
    } else if gcpercent >= 0 {                                                                                                                                                                              
        // Ensure there's always a little margin so that the                                                                                                                                                
        // mutator assist ratio isn't infinity.                                                                                                                                                             
        maxTriggerRatio := 0.95 * float64(gcpercent) / 100                                                                                                                                                  
        if triggerRatio > maxTriggerRatio {                                                                                                                                                                 
            triggerRatio = maxTriggerRatio                                                                                                                                                                  
        }                                                                                                                                                                                                   
    }                                                                                                                                                                                                       
    memstats.triggerRatio = triggerRatio

Consider the case where gcpercent == 3 as in the Go 1.14 log found here. If the input trigger ratio is < 0.6, then we'll set it at 0.6... but if it's greater, then we'll always set it to 0.95*3/100, or 0.0285. Given that gcpercent is 3, the latter is the correct behavior. In fact, when we see a 0.6 in a gcpacertrace, we also see a spike in heap use compared to Go 1.13. This is not good, since we're not respecting the trade-off.

I think the fix is to scale the minTriggerRatio just like the maxTriggerRatio, though it probably stops making sense to have a minimum at some point (it gets so close to 0 anyway).

We should fix this and probably backport it to Go 1.14. This breaks the promise of GOGC in some cases, which I think counts as a correctness bug?

CC @aclements @randall77

[gopherbot]

Change https://golang.org/cl/223937 mentions this issue: runtime: ensure minTriggerRatio never exceeds maxTriggerRatio

[mknyszek]

@pijng Would you be willing to try out https://golang.org/cl/223937 and to see if it helps?

[ianlancetaylor]

@gopherbot Please open a backport to 1.14. The comment above suggests that we should do this to fix GOGC in 1.14.

[gopherbot]

Backport issue(s) opened: #37928 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[ianlancetaylor]

@mknyszek Should this be for 1.14.1 or can it wait until 1.14.2?

[pijng]

@mknyszek I did the same tests as before.
It definitely helped and now I don't see any significant difference between 1.13.3 and 1.14 with https://golang.org/cl/223937.

Attaching log just in case.

If additional actions/logs are needed – I'm always ready.

[y3llowcake]

I believe we hit this issue in a recent upgrade of a high QPS production service to go 1.14. On startup, this service makes a few very large heap allocations and then dials down GOGC. During normal request processing operation it makes a high rate of smaller allocations. We found it is now OOMing where previously it did not.

I can write a constrained example program to demonstrate this behavior if it would help?

[mknyszek]

@y3llowcake that would be great! Let me know if you need any help; I think I've nailed down the conditions for reproducing this, but it's a little tricky to construct the right benchmark.

@ianlancetaylor After talking to some folks (and you), it looks like the 1.14.1 release is very close and we shouldn't delay it more. We'll get this in 1.14.2.

[y3llowcake]

I didn't get a chance to write up that program today, but figured I would at least share the snippet below. It's taken directly from the service in question and is the function we are using to adjust GOGC.

// This function attempts to set GOGC to a value that will maintain a maximum
// **additional** heap size of maxBytes. Call this after making a very large
// heap allocation.
func AdjustGCPercent(maxBytes uint64) (int, error) {
	if maxBytes < 1 {
		return 0, fmt.Errorf("adjust gcpercent: max bytes must be > 0")
	}
	// Force a GC to run so that memstats are up to date.
	runtime.GC()
	s := runtime.MemStats{}
	var diff uint64
	p := 100
	for ; p > 0; p-- {
		debug.SetGCPercent(p)
		runtime.ReadMemStats(&s)
		// https://golang.org/pkg/runtime/#MemStats
		// NextGC is the target heap size of the next GC cycle.
		//
		// The garbage collector's goal is to keep HeapAlloc ≤ NextGC.
		// At the end of each GC cycle, the target for the next cycle
		// is computed based on the amount of reachable data and the
		// value of GOGC.
		diff = s.NextGC - s.HeapAlloc
		if diff < maxBytes {
			break
		}
	}
	if diff < maxBytes {
		return p, nil
	}
	return p, fmt.Errorf("GC percent adjustment failed; last diff %d max %d", diff, maxBytes)
}