Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: test failed with out of memory on raspberrypi #3742

Closed
tjyang opened this issue Jun 15, 2012 · 16 comments
Closed

encoding/gob: test failed with out of memory on raspberrypi #3742

tjyang opened this issue Jun 15, 2012 · 16 comments

Comments

@tjyang
Copy link

@tjyang tjyang commented Jun 15, 2012

What steps will reproduce the problem?

 
1. increase runtime 120s to over 160s in run.bash file.
2. GOARM=5 ./all.bash

What is the expected output?
ok   encoding/gob    

What do you see instead?
ok      encoding/csv    0.198s
throw: runtime: out of memory

goroutine 25 [running]:
encoding/gob.(*Decoder).readMessage(0x107b3be0, 0x1000cf66, 0x1078e290)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/decoder.go:92 +0x70
encoding/gob.(*Decoder).recvMessage(0x107b3be0, 0x255a8, 0x10758000)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/decoder.go:84 +0x130
encoding/gob.(*Decoder).decodeTypeSequence(0x107b3be0, 0x1079b100, 0x160, 0x0)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/decoder.go:142 +0x68
encoding/gob.(*Decoder).DecodeValue(0x107b3be0, 0x114288, 0x1079b138, 0x160, 0x0, ...)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/decoder.go:206 +0x174
encoding/gob.(*Decoder).Decode(0x107b3be0, 0x114280, 0x1079b138, 0x107691e0, 0x107465a0,
...)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/decoder.go:185 +0x27c
encoding/gob.encFuzzDec(0x1079bc78, 0x113fc0, 0x1079bc50, 0x107283e0, 0x107341b0, ...)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/codec_test.go:1429 +0x348
encoding/gob.testFuzz(0x107a1580, 0x994c53f0, 0x126fc084, 0x64, 0x40081f98, ...)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/codec_test.go:1468 +0x290
encoding/gob.TestFuzzRegressions(0x107a1580, 0xe)
        /usbpub/home/gobuild/go/src/pkg/encoding/gob/codec_test.go:1457 +0xd0
testing.tRunner(0x107a1580, 0x203bbc, 0x0)
        /usbpub/home/gobuild/go/src/pkg/testing/testing.go:273 +0xac
created by testing.RunTests
        /usbpub/home/gobuild/go/src/pkg/testing/testing.go:349 +0x758

goroutine 1 [chan receive]:
testing.RunTests(0x10c00, 0x203ac0, 0x4d, 0x4d, 0x31901, ...)
        /usbpub/home/gobuild/go/src/pkg/testing/testing.go:350 +0x778
testing.Main(0x10c00, 0x203ac0, 0x4d, 0x4d, 0x200670, ...)
        /usbpub/home/gobuild/go/src/pkg/testing/testing.go:285 +0x70
main.main()
        /tmp/go-build345853870/encoding/gob/_test/_testmain.go:199 +0x88

goroutine 2 [syscall]:
created by runtime.main
        /usbpub/home/gobuild/go/src/pkg/runtime/proc.c:221

goroutine 3 [syscall]:
created by addtimer
        /usbpub/home/gobuild/go/src/pkg/runtime/ztime_linux_arm.c:72
FAIL    encoding/gob    0.408s


Which compiler are you using (5g, 6g, 8g, gccgo)?

gobuild@raspberrypi:~/go/src$ hg id
c868b808b7a7+ tip
gobuild@raspberrypi:~/go/src$

Which operating system are you using?

gobuild@raspberrypi:~/go/src$ uname -a
Linux raspberrypi 3.1.9+ #90 Wed Apr 18 18:23:05 BST 2012 armv6l GNU/Linux
gobuild@raspberrypi:~/go/src$


Which version are you using?  (run 'go version')


gobuild@raspberrypi:~/go/src$ go version
go version weekly.2012-03-27 +c868b808b7a7
gobuild@raspberrypi:~/go/src$


Extra info.

memory and swap info from top output.

Tasks:  66 total,   1 running,  65 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    190836k total,   102720k used,    88116k free,    11980k buffers
Swap:   102396k total,     9352k used,    93044k free,    67476k cached
@minux
Copy link
Member

@minux minux commented Jun 15, 2012

Comment 1:

TestFuzzRegressions test in encoding/gob is known to require lots of memory.
@robpike
Copy link
Contributor

@robpike robpike commented Jun 15, 2012

Comment 2:

I think the fuzz test should just be deleted. it causes more troubles than it addresses.
@dsymonds
Copy link
Member

@dsymonds dsymonds commented Jun 15, 2012

Comment 3:

I would like to leave it in. It's great for finding weird edge cases, and it has not
been problematic for ages. I think we should just disable it on low-memory systems, or
just make it use less RAM.
@robpike
Copy link
Contributor

@robpike robpike commented Jun 15, 2012

Comment 4:

But how do you tell whether to run it or not? I suggest instead we enable it only under
a flag that is off by default.
@dsymonds
Copy link
Member

@dsymonds dsymonds commented Jun 15, 2012

Comment 5:

A bunch of other tests have a
  if runtime.GOOS == "arm" { return }
at the top of them. That would be fine here too.
@robpike
Copy link
Contributor

@robpike robpike commented Jun 15, 2012

Comment 6:

I'm not convinced. It's possible this is just the 32-bit gc issue and that the same
things can happen on small-memory 32-bit systems.
@dsymonds
Copy link
Member

@dsymonds dsymonds commented Jun 15, 2012

Comment 7:

We could limit it to only 64-bit machines, then. It's still highly
valuable to have it there.
@minux
Copy link
Member

@minux minux commented Jun 17, 2012

Comment 8:

FYI, only pkg/net/multicast_test.go skip a test on linux/ARM.
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 11, 2012

Comment 9:

Here is some debug about the gob oom error
raspberrypi(~/go/src/pkg/encoding/gob) % go test encoding/gob -v -test.run=Fuzz
runtime.sysmap: 1048576 bytes at fixed address 0x10700000
throw: runtime: out of memory - could not mmap fixed
exit status 2
FAIL    encoding/gob    0.011s
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 11, 2012

Comment 10:

This isn't an Arm problem, it affects pretty much any 32bit system. Running this on my
pandaboard the amount of RSS used for the Fuzz test varies wildly, most likely due to GC
conservatism.
pando(~/go/src/pkg/encoding/gob) % /usr/bin/time -v ./gob.test -test.run=Fuzz 2>&1 |
grep Maximum\ resident\ set
        Maximum resident set size (kbytes): 17200
pando(~/go/src/pkg/encoding/gob) % /usr/bin/time -v ./gob.test -test.run=Fuzz 2>&1 |
grep Maximum\ resident\ set
        Maximum resident set size (kbytes): 13520
pando(~/go/src/pkg/encoding/gob) % /usr/bin/time -v ./gob.test -test.run=Fuzz 2>&1 |
grep Maximum\ resident\ set
        Maximum resident set size (kbytes): 1058880
pando(~/go/src/pkg/encoding/gob) % /usr/bin/time -v ./gob.test -test.run=Fuzz 2>&1 |
grep Maximum\ resident\ set
        Maximum resident set size (kbytes): 10848
pando(~/go/src/pkg/encoding/gob) % /usr/bin/time -v ./gob.test -test.run=Fuzz 2>&1 |
grep Maximum\ resident\ set
        Maximum resident set size (kbytes): 1058848
It turns out this is another manifestation of issue #3225. Oddly the final comment by rsc
says that TestFuzz is disabled in -short mode, and this is still true. So there must be
another test that is triggering the same behaviour, possibly TestFuzzRegressions.

Labels changed: added arch-arm, os-linux.

Owner changed to @davecheney.

Status changed to Accepted.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 13, 2012

Comment 11:

Digging in further (and adding a bit of swap), this fuzz regression test reliably needs
279mb of rss to run.
@dsymonds, is this expected, or does this sound like a problem ?
=== RUN TestFuzzRegressions
--- PASS: TestFuzzRegressions (0.11 seconds)
        codec_test.go:1469: seed=1328492090837718000 n=100 e=*float32
PASS
ok      encoding/gob    0.239s
        Command being timed: "go test -v -test.run=FuzzReg"
        User time (seconds): 20.28
        System time (seconds): 1.93
        Percent of CPU this job got: 87%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.37
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 279072
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 57929
        Voluntary context switches: 1181
        Involuntary context switches: 919
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
@dsymonds
Copy link
Member

@dsymonds dsymonds commented Jul 13, 2012

Comment 12:

I wouldn't be surprised by that. This is throwing random data at gob,
and some forms are going to trigger large allocations. The trick is to
make the gob package not allocate stupid amounts and cleanly reject
extreme inputs, which could form a DoS vector. So, yes, it sounds like
the test is doing an adequate job.
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 13, 2012

Comment 13:

Maybe we should tell the Pi folks they should attach some kind of swap device if they
want to compile from source. Swap is pretty much mandatory to get through the gc test
portion of the build.
@dsymonds
Copy link
Member

@dsymonds dsymonds commented Jul 13, 2012

Comment 14:

That seems reasonable to me. Or just tell them that they won't be able
to run 100% of the tests without it.
@robpike
Copy link
Contributor

@robpike robpike commented Jul 13, 2012

Comment 15:

I'd prefer to disable the fuzz test unless explicitly requested.
@robpike
Copy link
Contributor

@robpike robpike commented Jul 13, 2012

Comment 16:

This issue was closed by revision bbe6017.

Status changed to Fixed.

@tjyang tjyang added fixed labels Jul 13, 2012
@mikioh mikioh changed the title raspberrypi: encoding/gob test failed with out of memory encoding/gob: test failed with out of memory on raspberrypi Feb 26, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.