runtime: concurrent GC leads to 5x memory consumption

[mikioh]

go version devel +2ae7737 Sat Jul 11 14:36:33 2015 +0000 freebsd/386

# go test -v -run=Stress$
=== RUN   TestTraceStress
runtime: memory allocated by OS (2188484608) not in usable range [0x93100000,0x13100000)
runtime: out of memory: cannot allocate 65536-byte block (640802816 in use)
fatal error: out of memory
(snip)

[mikioh]

/CC @dvyukov

[dvyukov]

That's concurrent GC issue.

I've run:
$ GODEBUG=gctrace=1 ./pprof.test -test.run=TestTraceStress$ -test.cpu=1,1,1,1,.......,1

If I sprinkle runtime.GC calls in the test, in particular inside of the following loop:

    for i := 0; i < 1e3; i++ {
        _ = make([]byte, 1<<20)
    }

Memory usage stabilizes at ~68MB.

But if I remove the calls, and let the concurrent GC do its work, memory steadily grows to 240MB, then to 300MB, then to 340MB.

If I enable gctracing, then I see lines like this:

gc #1095 @60.193s 9%: 0.35+0.59+0.66+7.4+0.61 ms clock, 1.4+0.59+0+0/1.3/0.16+2.4 ms cpu, 4->125->2 MB, 4 MB goal, 10 P

GC is basically 30x off of the goal.

GC must not ever let heap grow above the GOGC goal. If it needs some reserve room, it should aim at 0.9 GOGC or so, but then cap the heap at GOGC if it fails to reclaim memory in time.

[RLH]

This is certainly worth looking into but stating that "GC must not ever let
heap grow above the GOGC goal" is promising something that Go can only
promise in an operational sense based on the current implementation. That
definition seems like a rat hole. Keeping GOGC a goal met by quality of
service setting makes more sense.

On Sun, Jul 19, 2015 at 5:15 PM, Dmitry Vyukov notifications@github.com
wrote:

That's concurrent GC issue.

I've run:
$ GODEBUG=gctrace=1 ./pprof.test -test.run=TestTraceStress$
-test.cpu=1,1,1,1,.......,1

If I sprinkle runtime.GC calls in the test, in particular inside of the
following loop:

for i := 0; i < 1e3; i++ {
    _ = make([]byte, 1<<20)
}

Memory usage stabilizes at ~68MB.

But if I remove the calls, and let the concurrent GC do its work, memory
steadily grows to 240MB, then to 300MB, then to 340MB.

If I enable gctracing, then I see lines like this:

gc #1095 #1095 @60.193s 9%:
0.35+0.59+0.66+7.4+0.61 ms clock, 1.4+0.59+0+0/1.3/0.16+2.4 ms cpu,
4->125->2 MB, 4 MB goal, 10 P

GC is basically 30x off of the goal.

GC must not ever let heap grow above the GOGC goal. If it needs some
reserve room, it should aim at 0.9 GOGC or so, but then cap the heap at
GOGC if it fails to reclaim memory in time.

—
Reply to this email directly or view it on GitHub
#11677 (comment).

[dvyukov]

If it is not a hard limit, than it is not possible to provision a machine for a Go program.