New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic("x") on corrupt db #155

Open
calmh opened this Issue Oct 6, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@calmh

calmh commented Oct 6, 2016

Got the panic below. Most likely the database is corrupt, but the presence of the panic looks like a debug thing so I figure you may want to know? Otherwise perhaps just return the error and we'll panic for you :)

panic: x

goroutine 1 [running]:
panic(0x99ad00, 0xc420042000)
/usr/pkg/go/src/runtime/panic.go:500 +0x1a1
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(Batch).decodeRec(0xc420402518, 0xc420402238, 0xc420402270, 0x496a)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/batch.go:210 +0x546
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(Batch).memReplay(0xc420402518, 0xc4200e00e0, 0xc420198000, 0xfae7)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/batch.go:246 +0x82
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(Batch).memDecodeAndReplay(0xc420402518, 0x88b0211, 0xc420198000, 0xfae7, 0x1fe00, 0xc4200e00e0, 0x0, 0x0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/batch.go:253 +0x7e
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(DB).recoverJournal(0xc42023c000, 0xc42023c000, 0xc4201c8ae0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:572 +0x914
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.openDB(0xc420093ad0, 0x0, 0x0, 0xc420093ad0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:124 +0x846
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.Open(0xe91520, 0xc4203e82a0, 0xc420080280, 0x0, 0x0, 0x0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:196 +0x115
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile(0xc420146090, 0x2c, 0xc420080280, 0xc420402820, 0x119ceb33, 0x708f1925e269b5ae)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:218 +0x97
github.com/syncthing/syncthing/lib/db.Open(0xc420146090, 0x2c, 0xa4efa6, 0x8, 0xc4200852a8)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/db/leveldb_dbinstance.go:50 +0x92
main.syncthingMain(0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xa4c6e4, 0x1, 0x0, 0x0, ...)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:657 +0xba5
main.main()
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:373 +0x239

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/pkg/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 7 [syscall]:
os/signal.signal_recv(0x0)
/usr/pkg/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/pkg/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/pkg/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 8 [chan receive]:
main.trackCPUUsage()
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:24 +0xf9
created by main.init.3
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:17 +0x35

goroutine 9 [select, locked to thread]:
runtime.gopark(0xc17ea0, 0x0, 0xa4e0e9, 0x6, 0x18, 0x2)
/usr/pkg/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc420021f30, 0x0, 0x18)
/usr/pkg/go/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc420021f30)
/usr/pkg/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
/usr/pkg/go/src/runtime/signal1_unix.go:304 +0x2d6
runtime.goexit()
/usr/pkg/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 10 [chan receive]:
main.setupSignalHandling.func1(0xc42005c8a0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:841 +0x40
created by main.setupSignalHandling
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:843 +0xfa

goroutine 11 [chan receive]:
main.setupSignalHandling.func2(0xc42005ca20)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:851 +0x40
created by main.setupSignalHandling
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:853 +0x1f5

goroutine 12 [select]:
github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(Supervisor).Serve(0xc4200e0380)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:338 +0xe61
created by github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(Supervisor).ServeBackground
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:297 +0x43

goroutine 13 [select]:
github.com/syncthing/syncthing/lib/events.(Subscription).Poll(0xc4200b9100, 0xdf8475800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:239 +0x3f8
github.com/syncthing/syncthing/lib/events.(bufferedSubscription).pollingLoop(0xc420053bd0)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:293 +0x55
created by github.com/syncthing/syncthing/lib/events.NewBufferedSubscription
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:287 +0x22f

goroutine 14 [select]:
github.com/syncthing/syncthing/lib/events.(Subscription).Poll(0xc4200b9160, 0xdf8475800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:239 +0x3f8
github.com/syncthing/syncthing/lib/events.(bufferedSubscription).pollingLoop(0xc420053c20)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:293 +0x55
created by github.com/syncthing/syncthing/lib/events.NewBufferedSubscription
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/lib/events/events.go:287 +0x22f

goroutine 18 [select]:
github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4200e0000)
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x247
created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/var/obj/pkg/net/syncthing/work/gopath/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x1bc
@abs0

This comment has been minimized.

Show comment
Hide comment
@abs0

abs0 Oct 6, 2016

This is from syncthing, and its index-v0.14.0.db is around 14GB. I can save a copy if needed.

abs0 commented Oct 6, 2016

This is from syncthing, and its index-v0.14.0.db is around 14GB. I can save a copy if needed.

@syndtr

This comment has been minimized.

Show comment
Hide comment
@syndtr

syndtr Oct 6, 2016

Owner

@calmh Good catch. This actually a piece of debugging code that gotten overlooked. However this quite interesting since corrupted records should not pass journal reader in the first place, I assume syncthing using default strict level correct?

@abs0 I only need the journal files, xxxxxx.log files. Thanks.

Owner

syndtr commented Oct 6, 2016

@calmh Good catch. This actually a piece of debugging code that gotten overlooked. However this quite interesting since corrupted records should not pass journal reader in the first place, I assume syncthing using default strict level correct?

@abs0 I only need the journal files, xxxxxx.log files. Thanks.

@abs0

This comment has been minimized.

Show comment
Hide comment
@abs0

abs0 Oct 6, 2016

Attached zip of log from main syncthing dir There are 43 other .log files in tmp-index-sorter. In case they are useful I've put all the logs (including the syncthing panic traces) up at http://sync.absd.org/issue155panic.txz for a while

184180.log.zip

abs0 commented Oct 6, 2016

Attached zip of log from main syncthing dir There are 43 other .log files in tmp-index-sorter. In case they are useful I've put all the logs (including the syncthing panic traces) up at http://sync.absd.org/issue155panic.txz for a while

184180.log.zip

@syndtr syndtr added the bug label Oct 9, 2016

syndtr added a commit that referenced this issue Oct 10, 2016

leveldb: partially rewrite batch and db writer
The new batch implementation is more cleaner by spliting batch header
buffer. Also added index, replaying batch should be much more performant now.

DB writer now guarantee that user batch content won't be changed, previous
implementation would append batch directly into user batch during write
merge, this plain wrong and can potentially cause issue. Also added options
to disable write merge.

There are internal corruption detected on issue #155, the batch apparently
contains invalid batch header and record with invalid length. Also lots
of batches with invalid sequence number present in the journal. Hopefully
this change fixes some of that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment