Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: unknown caller pc #48921

Open
zhaoyiping0622 opened this issue Oct 12, 2021 · 5 comments
Open

runtime: fatal error: unknown caller pc #48921

zhaoyiping0622 opened this issue Oct 12, 2021 · 5 comments

Comments

@zhaoyiping0622
Copy link

@zhaoyiping0622 zhaoyiping0622 commented Oct 12, 2021

Please file a new issue at golang.org/issue/new using this template:

What version of Go are you using (go version)?

$ go version
go version go1.16.8 linux/amd64

Does this issue reproduce with the latest release?

Not sure, before the crash occurred, the test has been running for two days, not sure whether it will reproduce. And no core file generated.

I'll try to run code in the latest release, but it may take some time.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/zhaoyiping/.cache/go-build"
GOENV="/home/zhaoyiping/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/zhaoyiping/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/zhaoyiping/go"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
GOROOT="/home/zhaoyiping/Applications/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/zhaoyiping/Applications/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.8"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/zhaoyiping/mit6.824/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build417777912=/tmp/go-build -gno-record-gcc-switches"
uname -sr: Linux 4.15.0-159-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.27-3ubuntu1.2) stable release version 2.27.
gdb --version: GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git

What did you do?

the code is here

I ran the following command in src/raft directory

go test -run 2C -race -failfast -count 1000 -timeout 0 > out1

and after 2 days with about 600+ rounds, I got the crash

the code did not use cgo and unsafe package.

What did you expect to see?

the program runs normally

What did you see instead?

the crash outputs:

Crash Output

unexpected fault address 0x600008140000
fatal error: fault
runtime: unexpected return pc for runtime.sigpanic called from 0x0
stack: frame={sp:0x7f67b09fed80, fp:0x7f67b09fedb8} stack=[0xc00004e800,0xc00004f000)

fatal error: unknown caller pc

runtime stack:
runtime.throw(0x9f81e9, 0x11)
/home/zhaoyiping/Applications/go/src/runtime/panic.go:1117 +0x72
runtime.gentraceback(0x46987b, 0x7f67b09fed60, 0x0, 0xc000102900, 0x0, 0x0, 0x7fffffff, 0x7f67b09fee50, 0x0, 0x0, ...)
/home/zhaoyiping/Applications/go/src/runtime/traceback.go:261 +0x1a56
runtime.addOneOpenDeferFrame.func1()
/home/zhaoyiping/Applications/go/src/runtime/panic.go:717 +0x91
runtime.systemstack(0x0)
/home/zhaoyiping/Applications/go/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
/home/zhaoyiping/Applications/go/src/runtime/proc.go:1246

goroutine 20 [running]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0x0
stack: frame={sp:0x7f67b09fec50, fp:0x7f67b09fec58} stack=[0xc00004e800,0xc00004f000)

runtime.systemstack_switch()
/home/zhaoyiping/Applications/go/src/runtime/asm_amd64.s:339 fp=0x7f67b09fec58 sp=0x7f67b09fec50 pc=0x4a50a0
created by 6.824/raft.TestMain
/home/zhaoyiping/mit6.824/src/raft/test_test.go:1104 +0x12f

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0014d9500, 0x9f6bd7, 0xe, 0xa11618, 0x1)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1239 +0x610
testing.runTests.func1(0xc0014d9500)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1511 +0xa7
testing.tRunner(0xc0014d9500, 0xc000225bc8)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1193 +0x203
testing.runTests(0xc000140558, 0xcb7360, 0x18, 0x18, 0x0, 0x0, 0x0, 0x203000)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1509 +0x613
testing.(*M).Run(0xc00019e600, 0x0)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1417 +0x3b4
6.824/raft.TestMain(0xc00019e600)
/home/zhaoyiping/mit6.824/src/raft/test_test.go:1121 +0x145
main.main()
_testmain.go:91 +0x272

goroutine 19 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f67e9158e88, 0x72, 0x72)
/home/zhaoyiping/Applications/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000024098, 0x72, 0x0, 0x0, 0x9f43ce)
/home/zhaoyiping/Applications/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6
internal/poll.(*pollDesc).waitRead(...)
/home/zhaoyiping/Applications/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000024080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/zhaoyiping/Applications/go/src/internal/poll/fd_unix.go:401 +0x32f
net.(*netFD).accept(0xc000024080, 0x47ddea, 0xc000102780, 0xc0002000a0)
/home/zhaoyiping/Applications/go/src/net/fd_unix.go:172 +0x58
net.(*TCPListener).accept(0xc00000e018, 0x4a417b, 0x161b29ff01701, 0x2591f166)
/home/zhaoyiping/Applications/go/src/net/tcpsock_posix.go:139 +0x50
net.(*TCPListener).Accept(0xc00000e018, 0x4a894c, 0x47722e, 0xc000065cd0, 0x49f6e0)
/home/zhaoyiping/Applications/go/src/net/tcpsock.go:261 +0x8e
net/http.(*Server).Serve(0xc000200000, 0xa8b3d0, 0xc00000e018, 0x0, 0x0)
/home/zhaoyiping/Applications/go/src/net/http/server.go:2961 +0x470
net/http.(*Server).ListenAndServe(0xc000200000, 0xd8, 0x0)
/home/zhaoyiping/Applications/go/src/net/http/server.go:2890 +0x10a
net/http.ListenAndServe(...)
/home/zhaoyiping/Applications/go/src/net/http/server.go:3144
6.824/raft.TestMain.func1()
/home/zhaoyiping/mit6.824/src/raft/test_test.go:1100 +0xf8
created by 6.824/raft.TestMain
/home/zhaoyiping/mit6.824/src/raft/test_test.go:1099 +0x87

goroutine 49348020 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc000593ec0, 0x966680, 0xc0005768d0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc000593ec0, 0xc0005768d0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x2, 0xc000593ec0, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348569 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc00024fec0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).syncLogLoop(0xc00000cb40, 0x2)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1199 +0x7d
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1213 +0xca

goroutine 49347637 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5a0, 0x9f7673, 0x10, 0x966880, 0xc00001e5a0, 0x9668c0, 0xc0006246f0, 0xc0002b3ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x0, 0xa8bef8, 0xc0004a4f00, 0xc00033fa50, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347730 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc000194f40, 0x9668c0, 0xc00053d340, 0xc0005c76c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc0004a5100, 0xc00033fa90, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347047 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc0007629c0, 0x966680, 0xc000400990, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc0007629c0, 0xc000400990, 0xc000350678)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x4, 0xc00024e180, 0xc0007629c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347567 [sleep]:
time.Sleep(0x1312d00)
/home/zhaoyiping/Applications/go/src/runtime/time.go:193 +0xd2
6.824/raft.(*config).one(0xc000370000, 0x9771e0, 0xa7dd60, 0x2, 0xc77201, 0x2)
/home/zhaoyiping/mit6.824/src/raft/config.go:540 +0x225
6.824/raft.TestPersist32C(0xc000138000)
/home/zhaoyiping/mit6.824/src/raft/test_test.go:707 +0x33c
testing.tRunner(0xc000138000, 0xa11618)
/home/zhaoyiping/Applications/go/src/testing/testing.go:1193 +0x203
created by testing.(*T).Run
/home/zhaoyiping/Applications/go/src/testing/testing.go:1238 +0x5d8

goroutine 49346999 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc0001940c0, 0x9668c0, 0xc00053c160, 0xc0004acec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc0004a5100, 0xc00033fa90, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348556 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e3e0, 0x9f7673, 0x10, 0x966880, 0xc00046a600, 0x9668c0, 0xc0000bcb00, 0xc00004fec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00050a3c0, 0x1, 0xa8bef8, 0xc0004a4f40, 0xc00033e830, 0x2, 0x2, 0x1, 0x1, 0xc00053f780)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348694 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cda60, 0x9f8413, 0x12, 0x986be0, 0xc000594cc0, 0x966680, 0xc00020cf30, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000cb40, 0x1, 0xc000594cc0, 0xc00020cf30, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00000cb40, 0x1, 0xc00024fd40, 0xc000594cc0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348575 [sleep]:
time.Sleep(0x3b9aca00)
/home/zhaoyiping/Applications/go/src/runtime/time.go:193 +0xd2
6.824/raft.Make.func1(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1332 +0xb8
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1329 +0x71d

goroutine 49348483 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e280, 0x9f8413, 0x12, 0x986be0, 0xc00086e420, 0x966680, 0xc0004409c0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00050a000, 0x2, 0xc00086e420, 0xc0004409c0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00050a000, 0x2, 0xc00086e420, 0xc0006203c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347941 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241fe0, 0x9f8413, 0x12, 0x986be0, 0xc0005c9d40, 0x966680, 0xc000440f60, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000c960, 0x4, 0xc0005c9d40, 0xc000440f60, 0x182e)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00000c960, 0x4, 0xc000b881e0, 0xc0005c9d40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347293 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc0002ca140, 0x9668c0, 0xc0004b3fd0, 0xc0005c66c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc0002e0e40, 0xc0000a6fb0, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348305 [chan receive]:
6.824/raft.(*config).applier(0xc000370000, 0x2, 0xc000564660)
/home/zhaoyiping/mit6.824/src/raft/config.go:162 +0x91
created by 6.824/raft.(*config).start1
/home/zhaoyiping/mit6.824/src/raft/config.go:283 +0xa06

goroutine 49348301 [chan receive]:
6.824/raft.(*Raft).ticker(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:913 +0xa6
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1325 +0x6ae

goroutine 49348576 [chan receive]:
6.824/raft.(*config).applier(0xc000370000, 0x0, 0xc000433380)
/home/zhaoyiping/mit6.824/src/raft/config.go:162 +0x91
created by 6.824/raft.(*config).start1
/home/zhaoyiping/mit6.824/src/raft/config.go:283 +0xa06

goroutine 49346936 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc000bef200, 0x966680, 0xc000440ff0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc000bef200, 0xc000440ff0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x4, 0xc00024e180, 0xc000bef200)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347185 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc00046bf40, 0x9668c0, 0xc0004b3940, 0xc00012f6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc0002e0e40, 0xc0000a6fb0, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347379 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e1c0, 0x9f7673, 0x10, 0x966880, 0xc0002ca460, 0x9668c0, 0xc0006242c0, 0xc0005c66c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c1e0, 0x2, 0xa8bef8, 0xc0002f7cc0, 0xc0003f6f60, 0x26, 0x3, 0xe, 0x25, 0xc000072d80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347594 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc00086e780, 0x966680, 0xc000566300, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc00086e780, 0xc000566300, 0xc8b)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc00086e780, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347886 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc0005fd320, 0x966680, 0xc000440ba0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc0005fd320, 0xc000440ba0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x2, 0xc0005fd320, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348624 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e3e0, 0x9f7673, 0x10, 0x966880, 0xc00001f300, 0x9668c0, 0xc0004ec330, 0xc0007a76c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00050a3c0, 0x1, 0xa8bef8, 0xc000520bc0, 0xc000269530, 0x3, 0x2, 0x1, 0x1, 0xc000125b80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347582 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc000c5e300, 0x966680, 0xc0005662d0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc000c5e300, 0xc0005662d0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc000c5e300)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347160 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002417e0, 0x9f7673, 0x10, 0x966880, 0xc0005fe6e0, 0x9668c0, 0xc000293810, 0xc000351ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c780, 0x4, 0xa8bef8, 0xc000259a80, 0xc00033e120, 0x26, 0x2, 0xd, 0x25, 0xc0004026c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348304 [sleep]:
time.Sleep(0x3b9aca00)
/home/zhaoyiping/Applications/go/src/runtime/time.go:193 +0xd2
6.824/raft.Make.func1(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1332 +0xb8
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1329 +0x71d

goroutine 49347938 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241f60, 0x9f8413, 0x12, 0x986be0, 0xc0005c97a0, 0x966680, 0xc0001294d0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000c960, 0x0, 0xc0005c97a0, 0xc0001294d0, 0x374)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00000c960, 0x0, 0xc0005c97a0, 0xc000b881e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347870 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002417e0, 0x9f7673, 0x10, 0x966880, 0xc00001eec0, 0x9668c0, 0xc0004ec460, 0xc00032d6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c780, 0x4, 0xa8bef8, 0xc000521680, 0xc00033f450, 0x29, 0x2, 0xd, 0x25, 0xc000072240)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348528 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e280, 0x9f8413, 0x12, 0x986be0, 0xc0006b70e0, 0x966680, 0xc000576ed0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00050a000, 0x2, 0xc0006b70e0, 0xc000576ed0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00050a000, 0x2, 0xc0006b70e0, 0xc0006203c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347724 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5c0, 0x9f7673, 0x10, 0x966880, 0xc00001eb40, 0x9668c0, 0xc0006250b0, 0xc0000516c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x1, 0xa8bef8, 0xc00015d940, 0xc0003d8d90, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347152 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc000837500, 0x966680, 0xc000441290, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc000837500, 0xc000441290, 0x5c5)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x3, 0xc000837500, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347726 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5e0, 0x9f7673, 0x10, 0x966880, 0xc000332880, 0x9668c0, 0xc0000bd790, 0xc00004e6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x2, 0xa8bef8, 0xc0002d8000, 0xc0003d8de0, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347823 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc0005fce40, 0x966680, 0xc000440b10, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc0005fce40, 0xc000440b10, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc0005fce40, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347342 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e220, 0x9f7673, 0x10, 0x966880, 0xc0002ca240, 0x9668c0, 0xc000624150, 0xc0005c66c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c1e0, 0x4, 0xa8bef8, 0xc0002e1400, 0xc0000a7320, 0x26, 0x3, 0xe, 0x25, 0xc000072d80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348677 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cda60, 0x9f8413, 0x12, 0x986be0, 0xc00025cc00, 0x966680, 0xc00020c660, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000cb40, 0x1, 0xc00025cc00, 0xc00020c660, 0x374)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00000cb40, 0x1, 0xc00025cc00, 0xc00024fd40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347860 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc000332ac0, 0x9668c0, 0xc0000bdff0, 0xc0007a76c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc000521dc0, 0xc000490040, 0x28, 0x4, 0xe, 0x25, 0xc0000722d0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347900 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241f60, 0x9f7673, 0x10, 0x966880, 0xc00001ef20, 0x9668c0, 0xc0004ec570, 0xc0002b2ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c960, 0x0, 0xa8bef8, 0xc0003395c0, 0xc000269e90, 0x2a, 0x3, 0xe, 0x25, 0xc000072750)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347939 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241fe0, 0x9f8413, 0x12, 0x986be0, 0xc0005c9860, 0x966680, 0xc000440f00, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000c960, 0x4, 0xc0005c9860, 0xc000440f00, 0xc000131e78)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00000c960, 0x4, 0xc0005c9860, 0xc000b881e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347977 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc000b88a80, 0x966680, 0xc000441620, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc000b88a80, 0xc000441620, 0x145c)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc000b88a80, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348295 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc000594a80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).sendHeartBeatLoop(0xc00050a3c0, 0x0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:977 +0x32b
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1212 +0x9e

goroutine 49347488 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc0006d9200, 0x966680, 0xc000566c60, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc0006d9200, 0xc000566c60, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x2, 0xc00024e180, 0xc0006d9200)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347494 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc00095d560, 0x966680, 0xc0004011a0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc00095d560, 0xc0004011a0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc00095d560)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347402 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5c0, 0x9f7673, 0x10, 0x966880, 0xc0005ff180, 0x9668c0, 0xc00053d6a0, 0xc00068d6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x1, 0xa8bef8, 0xc0002e0c80, 0xc0000a6f70, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347770 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc00025d860, 0x966680, 0xc0004409f0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc00025d860, 0xc0004409f0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x3, 0xc00024e180, 0xc00025d860)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347235 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e120, 0x9f7673, 0x10, 0x966880, 0xc00046bfc0, 0x9668c0, 0xc0004b3a00, 0xc0000516c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c1e0, 0x1, 0xa8bef8, 0xc0002e1200, 0xc0000a72e0, 0x26, 0x3, 0xe, 0x25, 0xc000072d80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347350 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc000568a20, 0x966680, 0xc000400e70, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc000568a20, 0xc000400e70, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc000568a20)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347328 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002414e0, 0x9f7673, 0x10, 0x966880, 0xc0005fea40, 0x9668c0, 0xc00053c900, 0xc000050ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c5a0, 0x4, 0xa8bef8, 0xc00043b180, 0xc00033eb10, 0x27, 0x1, 0xd, 0x25, 0xc000402c60)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347219 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5e0, 0x9f7673, 0x10, 0x966880, 0xc00046bf80, 0x9668c0, 0xc0004b39a0, 0xc00032cec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x2, 0xa8bef8, 0xc0002e1040, 0xc0000a6ff0, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348034 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc000c3b620, 0x966680, 0xc000567bf0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc000c3b620, 0xc000567bf0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x2, 0xc00024e180, 0xc000c3b620)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347665 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc0006b79e0, 0x966680, 0xc000128540, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc0006b79e0, 0xc000128540, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc0006b79e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347990 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc000c3b020, 0x966680, 0xc000441650, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc000c3b020, 0xc000441650, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x3, 0xc00024e180, 0xc000c3b020)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347311 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241460, 0x9f7673, 0x10, 0x966880, 0xc0002a1dc0, 0x9668c0, 0xc000019760, 0xc00034a6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c5a0, 0x0, 0xa8bef8, 0xc0001b7a00, 0xc0003d80b0, 0x27, 0x1, 0xd, 0x25, 0xc000402c60)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348298 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc00025d6e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).syncLogLoop(0xc00050a3c0, 0x1)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1204 +0x105
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1213 +0xca

goroutine 49347773 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc00025db00, 0x966680, 0xc000440a20, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc00025db00, 0xc000440a20, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x3, 0xc00025db00, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348299 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc000594ba0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).processAppendEntriesLoop(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1043 +0x309
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1216 +0xf6

goroutine 49347398 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5a0, 0x9f7673, 0x10, 0x966880, 0xc0002a1f20, 0x9668c0, 0xc000019e80, 0xc0005c66c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x0, 0xa8bef8, 0xc000259300, 0xc00057dd80, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347209 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5a0, 0x9f7673, 0x10, 0x966880, 0xc0002a1ca0, 0x9668c0, 0xc000019150, 0xc000330ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x0, 0xa8bef8, 0xc000259300, 0xc00057dd80, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347982 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e600, 0x9f7673, 0x10, 0x966880, 0xc0003336e0, 0x9668c0, 0xc0004213c0, 0xc00032d6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x3, 0xa8bef8, 0xc000521dc0, 0xc000490040, 0x28, 0x4, 0xe, 0x25, 0xc0000722d0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348696 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cda60, 0x9f8413, 0x12, 0x986be0, 0xc000594e40, 0x966680, 0xc00020cf60, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000cb40, 0x1, 0xc000594e40, 0xc00020cf60, 0x1a02)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00000cb40, 0x1, 0xc000594e40, 0xc00024fd40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347574 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5c0, 0x9f7673, 0x10, 0x966880, 0xc000332400, 0x9668c0, 0xc0000bc260, 0x10)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x1, 0xa8bef8, 0xc00015d940, 0xc0003d8d90, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347262 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc0004bbe60, 0x966680, 0xc000400b10, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc0004bbe60, 0xc000400b10, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc0004bbe60, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348572 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc00025d920)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).ticker(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:905 +0x1a5
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1325 +0x6ae

goroutine 49347668 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5c0, 0x9f7673, 0x10, 0x966880, 0xc0003327a0, 0x9668c0, 0xc0000bcf70, 0xc0007a36c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x1, 0xa8bef8, 0xc00015d940, 0xc0003d8d90, 0x27, 0x4, 0xe, 0x25, 0xc00071cf30)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347406 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241460, 0x9f7673, 0x10, 0x966880, 0xc0002a1f60, 0x9668c0, 0xc000019f60, 0xc0005c66c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c5a0, 0x0, 0xa8bef8, 0xc0001b7a00, 0xc0003d80b0, 0x27, 0x1, 0xd, 0x25, 0xc000402c60)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347568 [select]:
6.824/labrpc.MakeNetwork.func1(0xc000512140)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:153 +0x234
created by 6.824/labrpc.MakeNetwork
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:151 +0x27b

goroutine 49348300 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc000283aa0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).leaderCommitLogLoop(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1110 +0xeb
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1217 +0x118

goroutine 49348567 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc0005940c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).syncLogLoop(0xc00000cb40, 0x1)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1199 +0x7d
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1213 +0xca

goroutine 49347296 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5e0, 0x9f7673, 0x10, 0x966880, 0xc0002ca180, 0x9668c0, 0xc000624030, 0xc00004e6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x2, 0xa8bef8, 0xc0002e1040, 0xc0000a6ff0, 0x26, 0x4, 0xe, 0x25, 0xc000072cf0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347597 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc00086e840, 0x966680, 0xc000440030, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc00086e840, 0xc000440030, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x3, 0xc00086e840, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347836 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc0006faf00, 0x966680, 0xc000566990, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc0006faf00, 0xc000566990, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x4, 0xc00024e180, 0xc0006faf00)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347536 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc000241760, 0x9f7673, 0x10, 0x966880, 0xc0005ff320, 0x9668c0, 0xc00053df00, 0xc00012e6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c780, 0x0, 0xa8bef8, 0xc0004a5600, 0xc00033fea0, 0x28, 0x2, 0xd, 0x25, 0xc00071d170)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347258 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002417c0, 0x9f7673, 0x10, 0x966880, 0xc0005fe8a0, 0x9668c0, 0xc000293f40, 0xc0002b1ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c780, 0x3, 0xa8bef8, 0xc0002e1780, 0xc0000a7470, 0x26, 0x2, 0xd, 0x25, 0xc0004026c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348568 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc000594060)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).sendHeartBeatLoop(0xc00000cb40, 0x2)
/home/zhaoyiping/mit6.824/src/raft/raft.go:973 +0x145
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1212 +0x9e

goroutine 49348656 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cda60, 0x9f7673, 0x10, 0x966880, 0xc00001f3c0, 0x9668c0, 0xc0004ec590, 0xc00034bf88)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000cb40, 0x1, 0xa8bef8, 0xc000521880, 0xc000269b40, 0x4, 0x0, 0x2, 0x1, 0xc00053fd80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347059 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc00045a1e0, 0x966680, 0xc0004410e0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc00045a1e0, 0xc0004410e0, 0x1775)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc00045a1e0, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347771 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00016e5a0, 0x9f7673, 0x10, 0x966880, 0xc00001ec00, 0x9668c0, 0xc0006256a0, 0xc00004e6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c3c0, 0x0, 0xa8bef8, 0xc0003384c0, 0xc000269580, 0x28, 0x4, 0xe, 0x25, 0xc0000722d0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348302 [chan receive]:
6.824/raft.(*Raft).commit(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:343 +0x8f
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1326 +0x6d3

goroutine 49348571 [sleep]:
time.Sleep(0x989680)
/home/zhaoyiping/Applications/go/src/runtime/time.go:193 +0xd2
6.824/raft.(*Raft).leaderCommitLogLoop(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1106 +0x65
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1217 +0x118

goroutine 49347952 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc000c3ade0, 0x966680, 0xc0004415f0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc000c3ade0, 0xc0004415f0, 0x0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc000c3ade0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348638 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e3e0, 0x9f7673, 0x10, 0x966880, 0xc0002a1320, 0x9668c0, 0xc00053c690, 0xc00034e6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00050a3c0, 0x1, 0xa8bef8, 0xc000520bc0, 0xc000269530, 0x3, 0x2, 0x1, 0x1, 0xc000125b80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347684 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc0006b7bc0, 0x966680, 0xc000128570, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc0006b7bc0, 0xc000128570, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x1, 0xc0006b7bc0, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348480 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e280, 0x9f8413, 0x12, 0x986be0, 0xc0006b62a0, 0x966680, 0xc000576b40, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00050a000, 0x2, 0xc0006b62a0, 0xc000576b40, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00050a000, 0x2, 0xc0006203c0, 0xc0006b62a0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348570 [select]:
6.824/raft.(*Raft).processAppendEntriesLoop(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1034 +0x2d0
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1216 +0xf6

goroutine 49347300 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc00065e480, 0x966680, 0xc000400bd0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc00065e480, 0xc000400bd0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x2, 0xc00024e180, 0xc00065e480)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348297 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc0002839e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).sendHeartBeatLoop(0xc00050a3c0, 0x1)
/home/zhaoyiping/mit6.824/src/raft/raft.go:977 +0x32b
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1212 +0x9e

goroutine 49348566 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc000594000)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).sendHeartBeatLoop(0xc00000cb40, 0x1)
/home/zhaoyiping/mit6.824/src/raft/raft.go:973 +0x145
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1212 +0x9e

goroutine 49347743 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc0005fc120, 0x966680, 0xc0005668d0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc0005fc120, 0xc0005668d0, 0x52)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x2, 0xc0005fc120, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348690 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cda60, 0x9f8413, 0x12, 0x986be0, 0xc0005943c0, 0x966680, 0xc00020c630, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00000cb40, 0x1, 0xc0005943c0, 0xc00020c630, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00000cb40, 0x1, 0xc00024fd40, 0xc0005943c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348573 [chan receive]:
6.824/raft.(*Raft).commit(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:343 +0x8f
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1326 +0x6d3

goroutine 49347495 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001ccaa0, 0x9f8413, 0x12, 0x986be0, 0xc00095d6e0, 0x966680, 0xc000566bd0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x4, 0xc00095d6e0, 0xc000566bd0, 0x833)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x4, 0xc00095d6e0, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49347539 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002414e0, 0x9f7673, 0x10, 0x966880, 0xc0003320e0, 0x9668c0, 0xc0000ec780, 0xc00068d6c8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c5a0, 0x4, 0xa8bef8, 0xc00043b180, 0xc00033eb10, 0x27, 0x1, 0xd, 0x25, 0xc000402c60)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347965 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc0005939e0, 0x966680, 0xc000576870, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc0005939e0, 0xc000576870, 0x93c7a5)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).collectAppendEntriesAndReply.func1(0xc00046c780, 0x1, 0xc0005939e0, 0xc00024e180)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1133 +0xfd
created by 6.824/raft.(*Raft).collectAppendEntriesAndReply
/home/zhaoyiping/mit6.824/src/raft/raft.go:1130 +0x12b

goroutine 49348588 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc00056e3e0, 0x9f7673, 0x10, 0x966880, 0xc0002a0c80, 0x9668c0, 0xc00053c0a0, 0x4)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00050a3c0, 0x1, 0xa8bef8, 0xc0004a4f40, 0xc00033e830, 0x2, 0x2, 0x1, 0x1, 0xc00053f780)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49347153 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca40, 0x9f8413, 0x12, 0x986be0, 0xc0008379e0, 0x966680, 0xc0005668a0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x2, 0xc0008379e0, 0xc0005668a0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x2, 0xc00024e180, 0xc0008379e0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347355 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca60, 0x9f8413, 0x12, 0x986be0, 0xc000568d80, 0x966680, 0xc0004417a0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x3, 0xc000568d80, 0xc0004417a0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x3, 0xc00024e180, 0xc000568d80)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49347058 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc00045a060, 0x966680, 0xc0002effb0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc00045a060, 0xc0002effb0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc00045a060)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348303 [chan receive]:
6.824/raft.(*Raft).installSnapshotLoop(0xc00050a3c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1221 +0x71
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1327 +0x6f8

goroutine 49346798 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0001cca20, 0x9f8413, 0x12, 0x986be0, 0xc000773260, 0x966680, 0xc0002efdd0, 0x44122b)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendAppendEntries(0xc00046c780, 0x1, 0xc000773260, 0xc0002efdd0, 0x47ebfc)
/home/zhaoyiping/mit6.824/src/raft/raft.go:749 +0x1e6
6.824/raft.(*Raft).sendHeartBeatLoop.func1(0xc00046c780, 0x1, 0xc00024e180, 0xc000773260)
/home/zhaoyiping/mit6.824/src/raft/raft.go:964 +0xfd
created by 6.824/raft.(*Raft).sendHeartBeatLoop
/home/zhaoyiping/mit6.824/src/raft/raft.go:961 +0x127

goroutine 49348574 [chan receive]:
6.824/raft.(*Raft).installSnapshotLoop(0xc00000cb40)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1221 +0x71
created by 6.824/raft.Make
/home/zhaoyiping/mit6.824/src/raft/raft.go:1327 +0x6f8

goroutine 49347634 [chan receive]:
6.824/labrpc.(*ClientEnd).Call(0xc0002417e0, 0x9f7673, 0x10, 0x966880, 0xc00001e160, 0x9668c0, 0xc000624270, 0xc0002b3ec8)
/home/zhaoyiping/mit6.824/src/labrpc/labrpc.go:112 +0x3e5
6.824/raft.(*Raft).sendRequestVote(...)
/home/zhaoyiping/mit6.824/src/raft/raft.go:719
6.824/raft.(*Raft).getVotes(0xc00000c780, 0x4, 0xa8bef8, 0xc000496dc0, 0xc000269310, 0x28, 0x2, 0xd, 0x25, 0xc00071d170)
/home/zhaoyiping/mit6.824/src/raft/raft.go:386 +0x1a5
created by 6.824/raft.(*Raft).requestVote
/home/zhaoyiping/mit6.824/src/raft/raft.go:413 +0x15f

goroutine 49348296 [chan receive]:
6.824/raft.waitCtxDoneWithTimeout(0xa8bf68, 0xc0005949c0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:52 +0xd9
6.824/raft.(*Raft).syncLogLoop(0xc00050a3c0, 0x0)
/home/zhaoyiping/mit6.824/src/raft/raft.go:1204 +0x105
created by 6.824/raft.(*Raft).LeaderGoRoutines
/home/zhaoyiping/mit6.824/src/raft/raft.go:1213 +0xca
exit status 2
FAIL 6.824/raft 112168.010s

@toothrot toothrot changed the title fatal error: unknown caller pc runtime: fatal error: unknown caller pc Oct 12, 2021
@toothrot toothrot added this to the Backlog milestone Oct 12, 2021
@toothrot
Copy link
Contributor

@toothrot toothrot commented Oct 12, 2021

Noting #44096 as somewhat similar, though it may have been c-specific. Also #44503.

I'd be particularly interested if it reproduced in 1.17.2 as well.

@randall77
Copy link
Contributor

@randall77 randall77 commented Oct 12, 2021

This definitely looks strange:

runtime.gentraceback(0x46987b, 0x7f67b09fed60, 0x0, 0xc000102900, 0x0, 0x0, 0x7fffffff, 0x7f67b09fee50, 0x0, 0x0, ...)

That second argument looks like an OS thread stack, not a goroutine stack. Goroutine stacks should start with 0xc0.
The runtime is trying to add a defer record to code running on the OS thread stack. That seems bound to fail. Code running on the OS thread stack shouldn't be deferring anything.

The trackback doesn't contain the function that is initiating the defer. My guess is it would be raft/test_test.go:1115, because it is in the goroutine started here:

/home/zhaoyiping/mit6.824/src/raft/test_test.go:1104 +0x12f

It's a defer in a loop, which is a somewhat unusual case. I have no idea why that would cause this kind of failure though.

Not related to the bug, but note that this use of defer doesn't make a whole lot of sense. You want f closed after WriteTo is done, but the defer will wait until the return from the whole function, which is when the read from ctx.Done() succeeds. You might have many deferred records and open files before that happens.

@zhaoyiping0622
Copy link
Author

@zhaoyiping0622 zhaoyiping0622 commented Oct 13, 2021

Noting #44096 as somewhat similar, though it may have been c-specific. Also #44503.

I'd be particularly interested if it reproduced in 1.17.2 as well.

thanks, I'll try to run the code in 1.17.2

@zhaoyiping0622
Copy link
Author

@zhaoyiping0622 zhaoyiping0622 commented Oct 13, 2021

This definitely looks strange:

runtime.gentraceback(0x46987b, 0x7f67b09fed60, 0x0, 0xc000102900, 0x0, 0x0, 0x7fffffff, 0x7f67b09fee50, 0x0, 0x0, ...)

That second argument looks like an OS thread stack, not a goroutine stack. Goroutine stacks should start with 0xc0. The runtime is trying to add a defer record to code running on the OS thread stack. That seems bound to fail. Code running on the OS thread stack shouldn't be deferring anything.

The trackback doesn't contain the function that is initiating the defer. My guess is it would be raft/test_test.go:1115, because it is in the goroutine started here:

/home/zhaoyiping/mit6.824/src/raft/test_test.go:1104 +0x12f

It's a defer in a loop, which is a somewhat unusual case. I have no idea why that would cause this kind of failure though.

Not related to the bug, but note that this use of defer doesn't make a whole lot of sense. You want f closed after WriteTo is done, but the defer will wait until the return from the whole function, which is when the read from ctx.Done() succeeds. You might have many deferred records and open files before that happens.

Thanks, it is definitely a bug. But in fact the file is not created, so i think the program did not run into the branch which contains the defer, it may not relate to the panic.

@zhaoyiping0622
Copy link
Author

@zhaoyiping0622 zhaoyiping0622 commented Oct 24, 2021

Noting #44096 as somewhat similar, though it may have been c-specific. Also #44503.

I'd be particularly interested if it reproduced in 1.17.2 as well.

reproduced using go 1.17.2

go env output:

go env Output
                                                                                                                                             
$ go env                                                                                                                                                                                                    
GO111MODULE=""                                                                                                                                                                                              
GOARCH="amd64"                                                                                                                                                                                              
GOBIN=""                                                                                                                                                                                                    
GOCACHE="/home/zhaoyiping/.cache/go-build"                                                                                                                                                                  
GOENV="/home/zhaoyiping/.config/go/env"                                                                                                                                                                     
GOEXE=""                                                                                                                                                                                                    
GOEXPERIMENT=""                                                                                                                                                                                             
GOFLAGS=""                                                                                                                                                                                                  
GOHOSTARCH="amd64"                                                                                                                                                                                          
GOHOSTOS="linux"                                                                                                                                                                                            
GOINSECURE=""                     
GOMODCACHE="/home/zhaoyiping/go/pkg/mod"                            
GONOPROXY=""                                                                  
GONOSUMDB=""                                                                                              
GOOS="linux"                                                     
GOPATH="/home/zhaoyiping/go"                                                                                      
GOPRIVATE=""    
GOPROXY="https://goproxy.io,direct"                                                                                         
GOROOT="/home/zhaoyiping/Applications/go1.17.2"  
GOSUMDB="sum.golang.org"                                              
GOTMPDIR=""                                             
GOTOOLDIR="/home/zhaoyiping/Applications/go1.17.2/pkg/tool/linux_amd64"
GOVCS=""                                                                                     
GOVERSION="go1.17.2"                                         
GCCGO="gccgo"                                                                                                                                                                          
AR="ar"                                                                                      
CC="gcc"                                                           
CXX="g++"                                                           
CGO_ENABLED="1"
GOMOD="/home/zhaoyiping/mit6.824/src/go.mod"                                
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""                                     
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"                                       
CGO_LDFLAGS="-g -O2"                           
PKG_CONFIG="pkg-config"        
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build936101353=/tmp/go-build -gno-record-gcc-switches"
uname -sr: Linux 4.15.0-159-generic                                    
Distributor ID: Ubuntu                                
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic                                                        
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.27-3ubuntu1.2) stable release version 2.27.
gdb --version: GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git

the new code is here

the error message is here, start from line 2622.

the core dump file is too big (about 1.2GB), I don't know how to share it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants