Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: "out of memory" running fuzz tests on windows-amd64-longtest #52744

Closed
bcmills opened this issue May 6, 2022 · 4 comments
Closed

cmd/go: "out of memory" running fuzz tests on windows-amd64-longtest #52744

bcmills opened this issue May 6, 2022 · 4 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented May 6, 2022

go test proxy running at GOPROXY=http://127.0.0.1:54095/mod
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_fuzz_fuzztime (25.52s)
        script_test.go:260: 
            # There are no seed values, so 'go test' should finish quickly. (2.215s)
            # Fuzzing should exit 0 after fuzztime, even if timeout is short. (23.078s)
            > go test -timeout=3s -fuzz=FuzzFast -fuzztime=5s
            [stdout]
            warning: starting with empty corpus
            fuzz: elapsed: 2s, execs: 0 (0/sec), new interesting: 0 (total: 0)
            fuzz: elapsed: 5s, execs: 316564 (105009/sec), new interesting: 0 (total: 0)
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            runtime: failed to create new OS thread (have 42 already; errno=1455)
            fatal error: runtime.newosproc
            runtime: VirtualAlloc of 104857600 bytes failed with errno=1455
            fatal error: out of memory
            fatal error: out of memory allocating heap arena metadata
            exit status 2
            FAIL	fuzz	5.615s
            [stderr]
            go: failed to remove work dir: remove $WORK\tmp\go-build4214020448\b001\fuzz.test.exe: Access is denied.
            [exit status 1]
            FAIL: testdata\script\test_fuzz_fuzztime.txt:8: unexpected command failure
            
    --- FAIL: TestScript/test_fuzz_minimize_dirty_cov (34.01s)
        script_test.go:260: 
            # Test that minimization doesn't use dirty coverage snapshots when it
            # is unable to actually minimize the input. We do this by checking that
            # a expected value appears in the cache. If a dirty coverage map is used
            # (i.e. the coverage map generated during the last minimization step,
            # rather than the map provided with the initial input) then this value
            # is unlikely to appear in the cache, since the map generated during
            # the last minimization step should not increase the coverage. (29.273s)
            > [short] skip
            > [!fuzz-instrumented] skip
            > env GOCACHE=$WORK/gocache
            > go test -fuzz=FuzzCovMin -fuzztime=25s -test.fuzzcachedir=$GOCACHE/fuzz
            [stdout]
            warning: starting with empty corpus
            fuzz: elapsed: 0s, execs: 0 (0/sec), new interesting: 0 (total: 0)
            fuzz: elapsed: 3s, execs: 628152 (209180/sec), new interesting: 6 (total: 6)
            fuzz: elapsed: 6s, execs: 1371740 (248104/sec), new interesting: 6 (total: 6)
            fuzz: elapsed: 9s, execs: 1985965 (203944/sec), new interesting: 6 (total: 6)
            runtime: VirtualAlloc of 24576 bytes failed with errno=1455
            fatal error: out of memory
…
    --- FAIL: TestScript/test_fuzz_dup_cache (33.71s)
        script_test.go:260: 
            # This test checks that cached corpus loading properly handles duplicate entries (this can
            # happen when a f.Add value has a duplicate entry in the cached corpus.) Duplicate entries
            # should be discarded, and the rest of the cache should be loaded as normal. (29.547s)
            > env GOCACHE=$WORK/cache
            > env GODEBUG=fuzzdebug=1
            > mkdir -p $GOCACHE/fuzz/fuzztest/FuzzTarget
            > go run ./populate $GOCACHE/fuzz/fuzztest/FuzzTarget
            > go test -fuzz=FuzzTarget -fuzztime=10x .
            [stdout]
            FAIL	fuzztest [build failed]
            [stderr]
            # go/token
            runtime: VirtualAlloc of 32768 bytes failed with errno=1455
            fatal error: out of memory
…

greplogs -l -e 'FAIL: TestScript/test_fuzz.*\n(\s+.*)*fatal error: out of memory'
2022-05-05T01:32:18-b922055/windows-amd64-longtest

I wonder if the fuzz tests might actually be using too much memory — they might be more memory-intensive than most of the other tests by virtue of running the fuzzer.

On the other hand, these tests were also running for a long time. Maybe there's something wrong with the fuzzer that is causing them to be much more expensive than expected? (attn @golang/fuzzing)

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 6, 2022
@bcmills bcmills added this to the Go1.19 milestone May 6, 2022
@bcmills
Copy link
Member Author

bcmills commented May 6, 2022

Marking as release-blocker for Go 1.19 because this looks to me like a pretty serious problem with either the tests or the fuzzing engine.

  • test_fuzz_minimize_dirty_cov intentionally fuzzes for 25s, which seems excessive for coverage-guided fuzzing to minimize a trivial input.
  • test_fuzz_dup_cache ran for ~30s before it crashed in cmd/compile.
  • test_fuzz_fuzztime was running for 25s before it failed, although it only fuzzed for ~5½ seconds. That suggests to me that it (and likely test_fuzz_dup_cache too) was slowed down by the other test(s).

@bcmills bcmills assigned bcmills and rolandshoemaker and unassigned bcmills May 6, 2022
@bcmills bcmills added the fuzz Issues related to native fuzzing support label May 6, 2022
@gopherbot
Copy link

Change https://go.dev/cl/404634 mentions this issue: cmd/go: make TestScript/test_fuzz_minimize_dirty_cov use less memory

@rolandshoemaker
Copy link
Member

Suspect this is just because test_fuzz_minimize_dirty_cov was a bit overly aggressive in it's runtime, and was consuming a bit too much memory. CL 393660 reduces memory consumption a bit, but also fuzzing just uses a lot of memory for reasons.

@bcmills
Copy link
Member Author

bcmills commented May 6, 2022

Let's give that a try, then — if it's still a problem after that, we could consider some other options that trade off wall-time for memory footprint (like capping GOMAXPROCS to something small, or tossing in a semaphore in the fuzz-instrumented condition).

@golang golang locked and limited conversation to collaborators Jun 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
Status: No status
Development

No branches or pull requests

3 participants