Crash with "POTENTIAL DEADLOCK" message

[andrewarrow]

I've gotten this crash twice now. I restart and everything is fine. Anyone else seen this?

I0722 13:29:42.472383   23223 base.go:182] Final exchange rate: 10001
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 84448 lock 0xc00052aeb0
I0722 13:29:52.825029   23223 base.go:116] Refreshing exchange rate...
~/core/lib/blockchain.go:1495 lib.(*Blockchain).ProcessBlock { bc.ChainLock.Lock() } <<<<<
~/core/lib/server.go:1235 lib.(*Server)._handleBlock { _, isOrphan, err = srv.blockchain.ProcessBlock(blk, true) }
~/core/lib/server.go:1529 lib.(*Server)._handlePeerMessages { srv._handleBlock(serverMessage.Peer, msg) }
~/core/lib/server.go:1577 lib.(*Server).messageHandler { srv._handlePeerMessages(serverMessage) }

Have been trying to lock it again for more than 10m0s
goroutine 3172891 lock 0xc00052aeb0  

Here is what goroutine 84448 doing now
goroutine 84448 [runnable]:
github.com/bitclout/core/lib.DBGetProfileEntryForPKIDWithTxn.func1(0xc133d99500, 0x13e8, 0x13e8, 0x13e8, 0x0)
        /home/ec2-user/core/lib/db_utils.go:3867 +0xda
github.com/dgraph-io/badger/v3.(*Item).Value(0xc002306e10, 0xc06e4a51f8, 0x0, 0x0)
        /home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.0/iterator.go:112 +0x138
github.com/bitclout/core/lib.DBGetProfileEntryForPKIDWithTxn(0xc0e8450000, 0xc0f2818ea0, 0xb)
        /home/ec2-user/core/lib/db_utils.go:3867 +0x16c
github.com/bitclout/core/lib.DBGetProfileEntryForPKID.func1(0xc0e8450000, 0x1a588dc10eab0000, 0xc0e8450000)
        /home/ec2-user/core/lib/db_utils.go:3881 +0x3d
github.com/dgraph-io/badger/v3.(*DB).View(0xc000164900, 0xc06e4a52b0, 0x0, 0x0)
        /home/ec2-user/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.0/txn.go:808 +0x95
github.com/bitclout/core/lib.DBGetProfileEntryForPKID(0xc000164900, 0xc0f2818ea0, 0xc0f2818ea0)
        /home/ec2-user/core/lib/db_utils.go:3880 +0x6f
        /home/ec2-user/core/lib/db_utils.go:3880 +0x6f
github.com/bitclout/core/lib.(*UtxoView).GetProfileEntryForPKID(0xc0efa92370, 0xc0f2818ea0, 0x21)
        /home/ec2-user/core/lib/block_view.go:3277 +0x79
github.com/bitclout/core/lib.(*UtxoView).GetProfileEntryForPublicKey(0xc0efa92370, 0xc057fa9410, 0x21, 0x21, 0xc068dd4e00)
        /home/ec2-user/core/lib/block_view.go:3264 +0x86
github.com/bitclout/core/lib.(*UtxoView)._connectFollow(0xc0efa92370, 0xc06ebaa230, 0xc0331d3ca0, 0x10000af23, 0x180, 0x0, 0x0, 0x3, 0x4, 0x0, ...)
        /home/ec2-user/core/lib/block_view.go:4098 +0x1e8
github.com/bitclout/core/lib.(*UtxoView)._connectTransaction(0xc0efa92370, 0xc06ebaa230, 0xc0331d3ca0, 0x0, 0x1010000af23, 0x1, 0xc00, 0xc068dd4de0, 0x3, 0x4, ...)
        /home/ec2-user/core/lib/block_view.go:6219 +0x885
github.com/bitclout/core/lib.(*UtxoView).ConnectTransaction(0xc0efa92370, 0xc06ebaa230, 0xc0331d3ca0, 0x0, 0x10000af23, 0xc068dd4de0, 0x3, 0x4, 0x2d445, 0x2d368, ...)
        /home/ec2-user/core/lib/block_view.go:6147 +0x94
github.com/bitclout/core/lib.(*UtxoView).ConnectBlock(0xc0efa92370, 0xc0ebf91710, 0xc0eee5e000, 0xb4d, 0xc00, 0x1, 0xc00, 0x0, 0x0, 0x100000000000000, ...)
        /home/ec2-user/core/lib/block_view.go:6309 +0x213
github.com/bitclout/core/lib.(*Blockchain).ProcessBlock(0xc00052ae70, 0xc0ebf91710, 0xc0020c5e01, 0x0, 0x0, 0x0)
        /home/ec2-user/core/lib/blockchain.go:1765 +0xd1a
github.com/bitclout/core/lib.(*Server)._handleBlock(0xc00087fc80, 0xc000358480, 0xc0ebf91710)
        /home/ec2-user/core/lib/server.go:1235 +0x965
github.com/bitclout/core/lib.(*Server)._handlePeerMessages(0xc00087fc80, 0xc0ed066680)
        /home/ec2-user/core/lib/server.go:1529 +0x137
github.com/bitclout/core/lib.(*Server).messageHandler(0xc00087fc80)
        /home/ec2-user/core/lib/server.go:1577 +0x215
created by github.com/bitclout/core/lib.(*Server).Start
        /home/ec2-user/core/lib/server.go:1764 +0xbd
Other goroutines holding locks:
goroutine 131434 lock 0xc05353c4c0
~/core/third_party/github.com/sasha-s/go-deadlock/deadlock.go:119 go-deadlock.(*RWMutex).Lock { lock(m.mu.Lock, m) } <<<<<
~/core/lib/txindex.go:156 lib.(*TXIndex).Start.func1 { err := txi.Update() }

goroutine 84444 lock 0xc05eb6a180
~/core/lib/mempool.go:2203 lib.(*BitCloutMempool).RegenerateReadOnlyView { mp.mtx.RLock() } <<<<<
~/core/lib/mempool.go:2162 lib.(*BitCloutMempool).StartReadOnlyUtxoViewRegenerator.func1 { mp.RegenerateReadOnlyView() }

[tijno]

@andrewarrow Ive not seen it - and i just checked my logs from the last 2 days and no mention of "DEADLOCK"

Im not running core direct though - running backend in docker.

[maebeam]

This doesn't make a ton of sense to me at first glance. Badger locking could mean you're out of file descriptors. Have you tried increasing ulimit -n?

[andrewarrow]

i think I see the problem. I was trying to use a swap file vs. real ram. I'm closing this and making a new instance with more real memory.