raftLog.term Panic

[BramGruneir]

full stack trace

E0624 18:04:48.094631   62323 raft_transport.go:139] could not get address for node 3: gossip.go:214: unable to lookup descriptor for node: 3
E0624 18:04:48.394731   62323 raft_transport.go:139] could not get address for node 3: gossip.go:214: unable to lookup descriptor for node: 3
panic: requested entry at index is unavailable

goroutine 37 [running]:
github.com/coreos/etcd/raft.(*raftLog).term(0xc208146410, 0x2418e, 0x4ad58c0)
    /Users/bram/go/src/github.com/coreos/etcd/raft/log.go:216 +0x17c
github.com/coreos/etcd/raft.(*raftLog).matchTerm(0xc208146410, 0x2418e, 0x0, 0x42cbaa1)
    /Users/bram/go/src/github.com/coreos/etcd/raft/log.go:239 +0x32
github.com/coreos/etcd/raft.(*raftLog).maybeAppend(0xc208146410, 0x2418e, 0x0, 0x40, 0x0, 0x0, 0x0, 0x2418e, 0xc20856e600)
    /Users/bram/go/src/github.com/coreos/etcd/raft/log.go:74 +0x58
github.com/coreos/etcd/raft.(*raft).handleAppendEntries(0xc208414d80, 0x3, 0x200000002, 0x100000001, 0x15, 0x0, 0x2418e, 0x0, 0x0, 0x0, ...)
    /Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:657 +0x24f
github.com/coreos/etcd/raft.stepFollower(0xc208414d80, 0x3, 0x200000002, 0x100000001, 0x15, 0x0, 0x2418e, 0x0, 0x0, 0x0, ...)
    /Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:628 +0x3ba
github.com/coreos/etcd/raft.(*raft).Step(0xc208414d80, 0x3, 0x200000002, 0x100000001, 0x15, 0x0, 0x2418e, 0x0, 0x0, 0x0, ...)
    /Users/bram/go/src/github.com/coreos/etcd/raft/raft.go:487 +0x259
github.com/coreos/etcd/raft.(*multiNode).run(0xc20810a780)
    /Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:236 +0xd0b
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/bram/go/src/github.com/coreos/etcd/raft/multinode.go:56 +0xb6

goroutine 1 [select]:
github.com/cockroachdb/cockroach/server/cli.runStart(0x4f4b180, 0xc208040cd0, 0x0, 0x5)
    /Users/bram/go/src/github.com/cockroachdb/cockroach/server/cli/start.go:140 +0xdcc
github.com/spf13/cobra.(*Command).execute(0x4f4b180, 0xc208040c80, 0x5, 0x5, 0x0, 0x0)
    /Users/bram/go/src/github.com/spf13/cobra/command.go:495 +0x65c
github.com/spf13/cobra.(*Command).Execute(0x4f49440, 0x0, 0x0)
    /Users/bram/go/src/github.com/spf13/cobra/command.go:560 +0x18d
github.com/cockroachdb/cockroach/server/cli.Run(0xc20800a010, 0x6, 0x6, 0x0, 0x0)
    /Users/bram/go/src/github.com/cockroachdb/cockroach/server/cli/cli.go:118 +0x88
main.main()
    /Users/bram/go/src/github.com/cockroachdb/cockroach/main.go:48 +0x29d

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x4f89540)
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1034 +0x75
created by github.com/cockroachdb/cockroach/util/log.init·1
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:604 +0xcc

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 7 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 34 [select]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc2081460a0, 0xc20807c480, 0xc208042840, 0x0, 0x0)
    /Users/bram/go/src/github.com/cockroachdb/cockroach/gossip/client.go:130 +0x1668
github.com/cockroachdb/cockroach/gossip.func·001()
    /Users/bram/go/src/github.com/cockroachdb/cockroach/gossip/client.go:85 +0x1a6
github.com/cockroachdb/cockroach/util.func·001()
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51
created by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3

goroutine 10 [select]:
github.com/cockroachdb/cockroach/rpc.(*RemoteClockMonitor).MonitorRemoteOffsets(0xc20801d6a0, 0xc208042840)
    /Users/bram/go/src/github.com/cockroachdb/cockroach/rpc/clock_offset.go:158 +0x637
github.com/cockroachdb/cockroach/server.func·010()
    /Users/bram/go/src/github.com/cockroachdb/cockroach/server/server.go:109 +0x3a
github.com/cockroachdb/cockroach/util.func·001()
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:75 +0x51
created by github.com/cockroachdb/cockroach/util.(*Stopper).RunWorker
    /Users/bram/go/src/github.com/cockroachdb/cockroach/util/stopper.go:76 +0xe3

[tamird]

@bdarnell can you triage?

[tbg]

@bdarnell, looks like it might be time to close this one since so much time has passed and the information here doesn't give any hints as to how this can be reproduced. What do you think?