weed compact: out of memory

[juliensobrier]

Whenever I run weedfs compact against a 30GB file, weedfs fails with the error "fatal error: runtime: out of memory". But it is not running out of RAM (9GB free out of 32GB) and disk space (30GB+ left). Memory usage seems to be constant around 2GB. When it fails, the newly compacted .cpd file is always 17GB, I'm using the latest version (0.70) AMD64.

Full stack trace:
I0917 15:06:33 32317 volume.go:110] loading index file 1266.idx readonly false
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xdc4c60, 0x16)
/home/chris/apps/go/src/runtime/panic.go:530 +0x90
runtime.sysMap(0xc822160000, 0xc0a80000, 0x12800077a00, 0x1184738)
/home/chris/apps/go/src/runtime/mem_linux.go:206 +0x9b
runtime.(_mheap).sysAlloc(0x1169e40, 0xc0a80000, 0x0)
/home/chris/apps/go/src/runtime/malloc.go:429 +0x191
runtime.(_mheap).grow(0x1169e40, 0x60540, 0x0)
/home/chris/apps/go/src/runtime/mheap.go:651 +0x63
runtime.(_mheap).allocSpanLocked(0x1169e40, 0x6053d, 0x8b3518)
/home/chris/apps/go/src/runtime/mheap.go:553 +0x4f6
runtime.(_mheap).alloc_m(0x1169e40, 0x6053d, 0xffffff0100000000, 0x7fa21fffedd0)
/home/chris/apps/go/src/runtime/mheap.go:437 +0x119
runtime.(_mheap).alloc.func1()
/home/chris/apps/go/src/runtime/mheap.go:502 +0x41
runtime.systemstack(0x7fa21fffede8)
/home/chris/apps/go/src/runtime/asm_amd64.s:307 +0xab
runtime.(_mheap).alloc(0x1169e40, 0x6053d, 0x10100000000, 0x1)
/home/chris/apps/go/src/runtime/mheap.go:503 +0x63
runtime.largeAlloc(0xc0a79778, 0xc800000001, 0x7fa2262cbc78)
/home/chris/apps/go/src/runtime/malloc.go:766 +0xb3
runtime.mallocgc.func3()
/home/chris/apps/go/src/runtime/malloc.go:664 +0x33
runtime.systemstack(0xc82001aa00)
/home/chris/apps/go/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
/home/chris/apps/go/src/runtime/proc.go:1048

goroutine 1 [running]:
runtime.systemstack_switch()
/home/chris/apps/go/src/runtime/asm_amd64.s:245 fp=0xc8200ff790 sp=0xc8200ff788
runtime.mallocgc(0xc0a79778, 0xaac740, 0x400000001, 0x0)
/home/chris/apps/go/src/runtime/malloc.go:665 +0x9eb fp=0xc8200ff868 sp=0xc8200ff790
runtime.newarray(0xaac740, 0xc0a79778, 0xc8200ff998)
/home/chris/apps/go/src/runtime/malloc.go:798 +0xc9 fp=0xc8200ff8a8 sp=0xc8200ff868
runtime.makeslice(0xa94900, 0xc0a79778, 0xc0a79778, 0x0, 0x0, 0x0)
/home/chris/apps/go/src/runtime/slice.go:32 +0x165 fp=0xc8200ff8f8 sp=0xc8200ff8a8
github.com/chrislusf/seaweedfs/go/storage.(_Needle).ReadNeedleBody(0xc8200aea00, 0xc82016c0c8, 0x2, 0x4113ee788, 0xc0a79778, 0x0, 0x0)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/storage/needle_read_write.go:242 +0x1db fp=0xc8200ff9c8 sp=0xc8200ff8f8
github.com/chrislusf/seaweedfs/go/storage.ScanVolumeFile(0xd217f8, 0x1, 0x0, 0x0, 0xc8000004f2, 0x0, 0xc8200ffc10, 0xd21801, 0xc8200ffc28, 0x0, ...)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/storage/volume.go:333 +0x631 fp=0xc8200ffb38 sp=0xc8200ff9c8
github.com/chrislusf/seaweedfs/go/storage.(_Volume).copyDataAndGenerateIndexFile(0xc820188180, 0xc820790580, 0x8, 0xc820790588, 0x8, 0x0, 0x0)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/storage/volume_vacuum.go:89 +0x347 fp=0xc8200ffc68 sp=0xc8200ffb38
github.com/chrislusf/seaweedfs/go/storage.(*Volume).Compact(0xc820188180, 0x0, 0x0)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/storage/volume_vacuum.go:24 +0x24d fp=0xc8200ffd08 sp=0xc8200ffc68
main.runCompact(0x11599a0, 0xc8200700f0, 0x0, 0x0, 0x0)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/weed/compact.go:40 +0x1a9 fp=0xc8200ffdc8 sp=0xc8200ffd08
main.main()
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/weed/weed.go:78 +0x68d fp=0xc8200fff50 sp=0xc8200ffdc8
runtime.main()
/home/chris/apps/go/src/runtime/proc.go:188 +0x2b0 fp=0xc8200fffa0 sp=0xc8200fff50
runtime.goexit()
/home/chris/apps/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200fffa8 sp=0xc8200fffa0

goroutine 17 [syscall, 50 minutes, locked to thread]:
runtime.goexit()
/home/chris/apps/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 19 [chan receive, 1 minutes]:
github.com/chrislusf/seaweedfs/go/glog.(*loggingT).flushDaemon(0x1162360)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:883 +0x67
created by github.com/chrislusf/seaweedfs/go/glog.init.1
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:410 +0x297

goroutine 34 [select, 50 minutes]:
github.com/chrislusf/seaweedfs/go/stats.(*ServerStats).Start(0xc820186d40)
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/stats/stats.go:92 +0x464
created by github.com/chrislusf/seaweedfs/go/weed/weed_server.init.1
/home/chris/dev/workspace/home/gopath/src/github.com/chrislusf/seaweedfs/go/weed/weed_server/common.go:27 +0x50

goroutine 35 [syscall, 50 minutes]:
os/signal.signal_recv(0x0)
/home/chris/apps/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/home/chris/apps/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/home/chris/apps/go/src/os/signal/signal_unix.go:28 +0x37

[chrislusf]

This is because the file content is corrupted somehow.

Do you still have the .idx file? If so, I can try another way to compact the .dat file.

[juliensobrier]

yes, I still have the idx file. This is happening on 1 of the 4 weedfs volumes only. It was out of disk space for some time.

[chrislusf]

Please use the latest version and try this command:

weed compact -dir=. -volumeId=1266 -method=1