cmd/go: "out of memory" running fuzz tests on windows-amd64-longtest

[bcmills]

go test proxy running at GOPROXY=http://127.0.0.1:54095/mod
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_fuzz_fuzztime (25.52s)
        script_test.go:260: 
            # There are no seed values, so 'go test' should finish quickly. (2.215s)
            # Fuzzing should exit 0 after fuzztime, even if timeout is short. (23.078s)
            > go test -timeout=3s -fuzz=FuzzFast -fuzztime=5s
            [stdout]
            warning: starting with empty corpus
            fuzz: elapsed: 2s, execs: 0 (0/sec), new interesting: 0 (total: 0)
            fuzz: elapsed: 5s, execs: 316564 (105009/sec), new interesting: 0 (total: 0)
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            runtime: failed to create new OS thread (have 42 already; errno=1455)
            fatal error: runtime.newosproc
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            fatal error: out of memory allocating heap arena metadata
            exit status 2
            FAIL	fuzz	5.615s
            [stderr]
            go: failed to remove work dir: remove $WORK\tmp\go-build4214020448\b001\fuzz.test.exe: Access is denied.
            [exit status 1]
            FAIL: testdata\script\test_fuzz_fuzztime.txt:8: unexpected command failure
            
    --- FAIL: TestScript/test_fuzz_minimize_dirty_cov (34.01s)
        script_test.go:260: 
            # Test that minimization doesn't use dirty coverage snapshots when it
            # is unable to actually minimize the input. We do this by checking that
            # a expected value appears in the cache. If a dirty coverage map is used
            # (i.e. the coverage map generated during the last minimization step,
            # rather than the map provided with the initial input) then this value
            # is unlikely to appear in the cache, since the map generated during
            # the last minimization step should not increase the coverage. (29.273s)
            > [short] skip
            > [!fuzz-instrumented] skip
            > env GOCACHE=$WORK/gocache
            > go test -fuzz=FuzzCovMin -fuzztime=25s -test.fuzzcachedir=$GOCACHE/fuzz
            [stdout]
            warning: starting with empty corpus
            fuzz: elapsed: 0s, execs: 0 (0/sec), new interesting: 0 (total: 0)
            fuzz: elapsed: 3s, execs: 628152 (209180/sec), new interesting: 6 (total: 6)
            fuzz: elapsed: 6s, execs: 1371740 (248104/sec), new interesting: 6 (total: 6)
            fuzz: elapsed: 9s, execs: 1985965 (203944/sec), new interesting: 6 (total: 6)
            runtime: VirtualAlloc of 24576 bytes failed with errno=1455
            fatal error: out of memory
…
    --- FAIL: TestScript/test_fuzz_dup_cache (33.71s)
        script_test.go:260: 
            # This test checks that cached corpus loading properly handles duplicate entries (this can
            # happen when a f.Add value has a duplicate entry in the cached corpus.) Duplicate entries
            # should be discarded, and the rest of the cache should be loaded as normal. (29.547s)
            > env GOCACHE=$WORK/cache
            > env GODEBUG=fuzzdebug=1
            > mkdir -p $GOCACHE/fuzz/fuzztest/FuzzTarget
            > go run ./populate $GOCACHE/fuzz/fuzztest/FuzzTarget
            > go test -fuzz=FuzzTarget -fuzztime=10x .
            [stdout]
            FAIL	fuzztest [build failed]
            [stderr]
            # go/token
            runtime: VirtualAlloc of 32768 bytes failed with errno=1455
            fatal error: out of memory
…

greplogs -l -e 'FAIL: TestScript/test_fuzz.*\n(\s+.*)*fatal error: out of memory'
2022-05-05T01:32:18-b922055/windows-amd64-longtest

I wonder if the fuzz tests might actually be using too much memory — they might be more memory-intensive than most of the other tests by virtue of running the fuzzer.

On the other hand, these tests were also running for a long time. Maybe there's something wrong with the fuzzer that is causing them to be much more expensive than expected? (attn @golang/fuzzing)

[bcmills]

Marking as release-blocker for Go 1.19 because this looks to me like a pretty serious problem with either the tests or the fuzzing engine.

• test_fuzz_minimize_dirty_cov intentionally fuzzes for 25s, which seems excessive for coverage-guided fuzzing to minimize a trivial input.
• test_fuzz_dup_cache ran for ~30s before it crashed in cmd/compile.
• test_fuzz_fuzztime was running for 25s before it failed, although it only fuzzed for ~5½ seconds. That suggests to me that it (and likely test_fuzz_dup_cache too) was slowed down by the other test(s).

[gopherbot]

Change https://go.dev/cl/404634 mentions this issue: cmd/go: make TestScript/test_fuzz_minimize_dirty_cov use less memory

[rolandshoemaker]

Suspect this is just because test_fuzz_minimize_dirty_cov was a bit overly aggressive in it's runtime, and was consuming a bit too much memory. CL 393660 reduces memory consumption a bit, but also fuzzing just uses a lot of memory for reasons.

[bcmills]

Let's give that a try, then — if it's still a problem after that, we could consider some other options that trade off wall-time for memory footprint (like capping GOMAXPROCS to something small, or tossing in a semaphore in the fuzz-instrumented condition).