datastore used after close causing errors and panic for various commands

[djdv]

Version information:

go-ipfs version: 0.5.0-dev-7e7b76259-dirty
Repo version: 7
System version: amd64/solaris
Golang version: go1.12.9

Description:

I'm using a modified version of 7e7b762 that simply adds in missing constant values to github.com/libp2p/go-reuseport/control_unix.go (const SO_REUSEPORT = 0x2004 and
const SO_REUSEADDR = 0x0004), and changes badger to use unix.Madvise instead of syscall.Madvise

I managed to run 2 instances of this build without any problems, pinning ~300GB of data from 1 instance to another.

However, running ipfs id and some other commands, while the daemon is not running, is causing issues, and sometimes panic. No issues while the daemon is up.
Log below:

The Illumos Project     SunOS 5.11      illumos-793ac993be      September 2019
dd@sun-storage:~$ ipfs id
{
        "ID": "QmWABRCqWoYCGyGV2PZ2T3D9upV39yMgkmFCprXVsKvAZr",
        "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDfb2SC+B7S/+494kcjMLl2yGjqXSF1vCzuQ5+vxzkHyEnOhX8/7cxHX6Y/HVZjcBA1IsbRdq1VRfg7yoO2PLh4JixwqcdBtCtFtfMXZVoF0Spz28HVPdrGK4z26j84+0l+BC5iOk6pSt1JbCJ2UK3SNWseZWPnahfiOfqYsevUdnz/19JV/Zls0J0VY4XRMvdo5cYBX4UqNJDjxw/Sm2IS1PmhWm3zNtjHTMw0ia+nrS60py92bAtswC/+qkaAkaZgBoilwQcqyNvAA5QgLaYr9G6J3oiMmR5IMFuDVjIKamZ1fvgBJ4pHNa+87WuEBNVwEeI9UL9LbxpAQcNNE7M3AgMBAAE=",
        "Addresses": null,
        "AgentVersion": "go-ipfs/0.5.0-dev/7e7b76259-dirty",
        "ProtocolVersion": "ipfs/0.1.0"
}
08:25:00.704 ERROR provider.q: error deleting queue entry with key (), due to error (Writes are blocked, possibly due to DropAll or Close), stopping provider queue.go:95
dd@sun-storage:~$ ipfs id
{
        "ID": "QmWABRCqWoYCGyGV2PZ2T3D9upV39yMgkmFCprXVsKvAZr",
        "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDfb2SC+B7S/+494kcjMLl2yGjqXSF1vCzuQ5+vxzkHyEnOhX8/7cxHX6Y/HVZjcBA1IsbRdq1VRfg7yoO2PLh4JixwqcdBtCtFtfMXZVoF0Spz28HVPdrGK4z26j84+0l+BC5iOk6pSt1JbCJ2UK3SNWseZWPnahfiOfqYsevUdnz/19JV/Zls0J0VY4XRMvdo5cYBX4UqNJDjxw/Sm2IS1PmhWm3zNtjHTMw0ia+nrS60py92bAtswC/+qkaAkaZgBoilwQcqyNvAA5QgLaYr9G6J3oiMmR5IMFuDVjIKamZ1fvgBJ4pHNa+87WuEBNVwEeI9UL9LbxpAQcNNE7M3AgMBAAE=",
        "Addresses": null,
        "AgentVersion": "go-ipfs/0.5.0-dev/7e7b76259-dirty",
        "ProtocolVersion": "ipfs/0.1.0"
}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xd71747]
goroutine 62 [running]:
github.com/dgraph-io/badger/skl.(*Skiplist).IncrRef(...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/dgraph-io/badger/skl/skl.go:86
github.com/dgraph-io/badger.(*DB).getMemTables(0xc00010d880, 0x0, 0x0, 0x0, 0x0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/dgraph-io/badger/db.go:481 +0xd7
github.com/dgraph-io/badger.(*Txn).NewIterator(0xc0038ba000, 0x1, 0x64, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/dgraph-io/badger/iterator.go:407 +0x75
github.com/ipfs/go-ds-badger.(*txn).query(0xc003698180, 0xc000036fa0, 0x12, 0x0, 0x0, 0x0, 0xc0002f5cd0, 0x1, 0x1, 0x1, ...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ds-badger/datastore.go:472 +0x184
github.com/ipfs/go-ds-badger.(*Datastore).Query(0xc000372880, 0xc000036fa0, 0x12, 0x0, 0x0, 0x0, 0xc0002f5cd0, 0x1, 0x1, 0x1, ...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ds-badger/datastore.go:245 +0x125
github.com/ipfs/go-ds-measure.(*measure).Query(0xc0000c6900, 0xc000036fa0, 0x12, 0x0, 0x0, 0x0, 0xc0002f5cd0, 0x1, 0x1, 0x1, ...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ds-measure/measure.go:196 +0x10a
github.com/ipfs/go-ds-measure.(*measure).Query(0xc0000c6b40, 0xc000036fa0, 0x12, 0x0, 0x0, 0x0, 0xc0002f5cd0, 0x1, 0x1, 0x1, ...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ds-measure/measure.go:196 +0x10a
github.com/ipfs/go-datastore/keytransform.(*Datastore).Query(0xc0036547e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0002f5cd0, 0x1, 0x1, 0x1, ...)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-datastore/keytransform/keytransform.go:66 +0x1d5
github.com/ipfs/go-ipfs-provider/queue.(*Queue).getQueueHead(0xc003668460, 0x0, 0x0, 0x0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ipfs-provider/queue/queue.go:137 +0x11e
github.com/ipfs/go-ipfs-provider/queue.(*Queue).work.func1(0xc003668460)
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ipfs-provider/queue/queue.go:83 +0x802
created by github.com/ipfs/go-ipfs-provider/queue.(*Queue).work
        src/github.com/ipfs/go-ipfs/vendor/github.com/ipfs/go-ipfs-provider/queue/queue.go:73 +0x3f
dd@sun-storage:~$ ipfs id
{
        "ID": "QmWABRCqWoYCGyGV2PZ2T3D9upV39yMgkmFCprXVsKvAZr",
        "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDfb2SC+B7S/+494kcjMLl2yGjqXSF1vCzuQ5+vxzkHyEnOhX8/7cxHX6Y/HVZjcBA1IsbRdq1VRfg7yoO2PLh4JixwqcdBtCtFtfMXZVoF0Spz28HVPdrGK4z26j84+0l+BC5iOk6pSt1JbCJ2UK3SNWseZWPnahfiOfqYsevUdnz/19JV/Zls0J0VY4XRMvdo5cYBX4UqNJDjxw/Sm2IS1PmhWm3zNtjHTMw0ia+nrS60py92bAtswC/+qkaAkaZgBoilwQcqyNvAA5QgLaYr9G6J3oiMmR5IMFuDVjIKamZ1fvgBJ4pHNa+87WuEBNVwEeI9UL9LbxpAQcNNE7M3AgMBAAE=",
        "Addresses": null,
        "AgentVersion": "go-ipfs/0.5.0-dev/7e7b76259-dirty",
        "ProtocolVersion": "ipfs/0.1.0"
}
08:25:31.428 ERROR provider.q: error deleting queue entry with key (), due to error (Writes are blocked, possibly due to DropAll or Close), stopping provider queue.go:95
dd@sun-storage:~$

This is happening for a variety of other commands as well, such as ipfs key list

ipfs key list -l
QmWABRCqW... self
12D3KooWND... ipfs-mount-bins
12D3KooWGY... testkey
14:58:01.705 ERROR provider.q: error deleting queue entry with key (), due to error (Writes are blocked, possibly due to DropAll or Close), stopping provider queue.go:95
dd@sun-storage:~/go/src/github.com/ipfs/go-ipfs$ ipfs key list
self
ipfs-mount-bins
testkey
14:58:51.441 ERROR provider.q: error deleting queue entry with key (), due to error (Writes are blocked, possibly due to DropAll or Close), stopping provider queue.go:95

randomly cc'ing @marten-seemann as the other Solaris user I know.