Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(server): 完成reactor server开发，但测试过程中遇到个问题 #26

Merged
merged 15 commits into from
Jun 28, 2024
Merged

feat(server): 完成reactor server开发，但测试过程中遇到个问题 #26

merged 15 commits into from
Jun 28, 2024

Conversation

Trinoooo
Copy link
Owner

@RocooHash 我最近搓了一个reactor服务器，但测试过程中遇到一个难以解决的问题，于是寄希望于求助你：）
问题描述：
当我执行测试用例时，会出现下文所示调试信息。信息指出socket在server端被不知名原因关闭（这是我能定位到的问题，但不一定准确）

2024/06/19 23:45:17 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:87: reactor #2 ready to register event, wrapper fd: 95
2024/06/19 23:45:17 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:87: reactor #0 ready to register event, wrapper fd: 111
2024/06/19 23:45:17 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:101: bad file descriptor [{111 -1 21 0xc000030180}] 127.0.0.1:63604 127.0.0.1:9999
2024/06/19 23:45:17 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:10···1: bad file descriptor [{95 -1 21 0xc00051c070}] 127.0.0.1:63596 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:63596->127.0.0.1:9999: read: connection reset by peer
    server_test.go:96: read tcp 127.0.0.1:63604->127.0.0.1:9999: read: connection reset by peer

另一个用于佐证的证据是goroutine pprof快照，可以看到reactor都阻塞在select，waitor都阻塞在syscall，dispatcher阻塞在connectons接收：

goroutine 519 [running]:
runtime/pprof.writeGoroutineStacks({0x100bfaf40, 0xc000056000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/pprof/pprof.go:703 +0x74
runtime/pprof.writeGoroutine({0x100bfaf40, 0xc000056000}, 0x2)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/pprof/pprof.go:692 +0x44
runtime/pprof.(*Profile).WriteTo(0x100e53060, {0x100bfaf40, 0xc000056000}, 0xc?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/pprof/pprof.go:329 +0x138
net/http/pprof.handler.ServeHTTP({0xc0000262b1, 0x9}, {0x100bfd0c8, 0xc000056000}, 0xc00013d6a0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/pprof/pprof.go:267 +0x4a8
net/http/pprof.Index({0x100bfd0c8?, 0xc000056000}, 0xc000220000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/pprof/pprof.go:384 +0xe4
net/http.HandlerFunc.ServeHTTP(0x100bf7758, {0x100bfd0c8, 0xc000056000}, 0x1009cf6f4?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:2136 +0x4c
net/http.(*ServeMux).ServeHTTP(0x0?, {0x100bfd0c8, 0xc000056000}, 0xc000220000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:2514 +0xa0
net/http.serverHandler.ServeHTTP({0xc0000c4060?}, {0x100bfd0c8, 0xc000056000}, 0xc000220000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:2938 +0x29c
net/http.(*conn).serve(0xc00040e120, {0x100bfd548, 0xc0002f2300})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:2009 +0x8f8
created by net/http.(*Server).Serve in goroutine 22
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3086 +0x63c

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc000183040, {0x100a8894b, 0x11}, 0x100bf6a20)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1649 +0x608
testing.runTests.func1(0x0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x84
testing.tRunner(0xc000183040, 0xc000207a58)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b4
testing.runTests(0xc0002348c0?, {0x100e578e0, 0x4, 0x4}, {0x0?, 0xc000207b7c?, 0x100e5eba0?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x6e8
testing.(*M).Run(0xc0002348c0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0x9f0
github.com/Trinoooo/eggie_kv/storage/server.TestMain(0x0?)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:27 +0x54
main.main()
	_testmain.go:61 +0x298

goroutine 21 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x10239bae0, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc0001f32a0, 0x10069f901?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001f3280)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:611 +0x314
net.(*netFD).accept(0xc0001f3280)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_unix.go:172 +0x38
net.(*TCPListener).accept(0xc0001ec8e0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock_posix.go:152 +0x3c
net.(*TCPListener).Accept(0xc0001ec8e0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock.go:315 +0x68
github.com/Trinoooo/eggie_kv/storage/server.(*ReactorServer).Serve(0xc0001b8240)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:332 +0x228
github.com/Trinoooo/eggie_kv/storage/server.TestReactorServer(0xc0001836c0)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:181 +0x214
testing.tRunner(0xc0001836c0, 0x100bf6a20)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b4
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5ec

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0x10239b9e8, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc0003060a0, 0x10069f901?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000306080)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:611 +0x314
net.(*netFD).accept(0xc000306080)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_unix.go:172 +0x38
net.(*TCPListener).accept(0xc0002f6060)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock_posix.go:152 +0x3c
net.(*TCPListener).Accept(0xc0002f6060)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock.go:315 +0x68
net/http.(*Server).Serve(0xc0002f0000, {0x100bfcf18, 0xc0002f6060})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3056 +0x44c
net/http.(*Server).ListenAndServe(0xc0002f0000)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:2985 +0xb4
net/http.ListenAndServe(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3239
github.com/Trinoooo/eggie_kv/storage/server.TestReactorServer.func1()
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:170 +0xd8
created by github.com/Trinoooo/eggie_kv/storage/server.TestReactorServer in goroutine 21
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:169 +0x30

goroutine 23 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc000388098?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0xc000388090)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/sync/waitgroup.go:116 +0x7c
github.com/Trinoooo/eggie_kv/storage/server.mockClient(0xc0001836c0, 0xc0001a2b70)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:47 +0x334
created by github.com/Trinoooo/eggie_kv/storage/server.TestReactorServer in goroutine 21
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:180 +0x20c

goroutine 24 [select, 9 minutes]:
github.com/Trinoooo/eggie_kv/storage/server.(*reactor).run(0xc00023c240)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:63 +0x30c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc0001a6140, 0xc0001ec960)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 21
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 25 [chan receive, 9 minutes]:
github.com/Trinoooo/eggie_kv/storage/server.(*dispatcher).run(0xc0001a2b30)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:296 +0x150
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc0001a6148, 0xc0001ec900)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 21
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 26 [select, 9 minutes]:
github.com/Trinoooo/eggie_kv/storage/server.(*reactor).run(0xc00023c1e0)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:63 +0x30c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc0001a6150, 0xc0001ec920)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 21
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 27 [select, 9 minutes]:
github.com/Trinoooo/eggie_kv/storage/server.(*reactor).run(0xc00023c210)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:63 +0x30c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc0001a6158, 0xc0001ec940)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 21
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 28 [syscall, 9 minutes]:
syscall.syscall6(0xc000097b48?, 0x1006760f4?, 0x8f?, 0x50?, 0xc000097b78?, 0x1006703a8?, 0xc000097bc8?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.kevent(0xc000097bd8?, 0x0, 0xc0001f5f3f?, 0xc0000b0280, 0xc0001f5e00?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:271 +0x78
syscall.Kevent(0xc00059c000?, {0x0, 0x0, 0x100e5eba0?}, {0xc0000b0280, 0xa, 0x10083c120?}, 0xc0001aa360?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:431 +0x74
github.com/Trinoooo/eggie_kv/storage/server.(*kqueuePoller).wait(0xc0001a6120, {0xc0001f5e00, 0xa, 0xc000097ea8?})
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:209 +0x1c0
github.com/Trinoooo/eggie_kv/storage/server.(*waiter).run(0xc0001ec880)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:146 +0x2a4
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc0001a6160, 0xc0001ec980)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 26
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 50 [syscall, 9 minutes]:
syscall.syscall6(0xc00009db48?, 0x1006760f4?, 0xc00009db48?, 0x100676604?, 0xc00009db58?, 0x10067d994?, 0xc00009dbc8?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.kevent(0xc00009dbd8?, 0x0, 0xc0000b013f?, 0xc0003a1cc0, 0xc00009dbf8?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:271 +0x78
syscall.Kevent(0xc000601500?, {0x0, 0x0, 0x100e5eba0?}, {0xc0003a1cc0, 0xa, 0x10083c120?}, 0xc0001aa360?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:431 +0x74
github.com/Trinoooo/eggie_kv/storage/server.(*kqueuePoller).wait(0xc0001a6130, {0xc0000b0000, 0xa, 0xc00009dea8?})
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:209 +0x1c0
github.com/Trinoooo/eggie_kv/storage/server.(*waiter).run(0xc0001ec8c0)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:146 +0x2a4
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc000090008, 0xc0000ac000)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 24
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 66 [syscall, 9 minutes]:
syscall.syscall6(0xc000115b48?, 0x1006760f4?, 0x8f?, 0x50?, 0xc000115b78?, 0x1006703a8?, 0xc000115bc8?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.kevent(0xc000115bd8?, 0x0, 0xc0003a013f?, 0xc0003592c0, 0xc0003a0000?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:271 +0x78
syscall.Kevent(0xc00016e700?, {0x0, 0x0, 0x100e5eba0?}, {0xc0003592c0, 0xa, 0x10083c120?}, 0xc0001aa360?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:431 +0x74
github.com/Trinoooo/eggie_kv/storage/server.(*kqueuePoller).wait(0xc0001a6128, {0xc0003a0000, 0xa, 0xc000115ea8?})
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:209 +0x1c0
github.com/Trinoooo/eggie_kv/storage/server.(*waiter).run(0xc0001ec8a0)
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:146 +0x2a4
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1.1(0xc00039c000, 0xc00039a000)
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:69 +0x9c
github.com/bytedance/gopkg/util/gopool.(*worker).run.func1()
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:70 +0x1b0
created by github.com/bytedance/gopkg/util/gopool.(*worker).run in goroutine 27
	/Users/sujun/go/pkg/mod/github.com/bytedance/gopkg@v0.0.0-20240507064146-197ded923ae3/util/gopool/worker.go:41 +0x88

goroutine 70 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x10239b038, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc0001360a0, 0xc0005aa3b8?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000136080, {0xc0005aa3b8, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000136080, {0xc0005aa3b8, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc000404018, {0xc0005aa3b8, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x8c
github.com/Trinoooo/eggie_kv/storage/server.longConnection(0xc0001836c0, {0xc000388088, 0x8, 0x8})
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:94 +0x240
github.com/Trinoooo/eggie_kv/storage/server.mockClient.func2()
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:44 +0xa0
created by github.com/Trinoooo/eggie_kv/storage/server.mockClient in goroutine 23
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:42 +0x14c

goroutine 520 [IO wait]:
internal/poll.runtime_pollWait(0x129be1140, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc000136120, 0xc00007e000?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000136100, {0xc00007e000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000136100, {0xc00007e000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc0001a6050, {0xc00007e000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x8c
net/http.(*connReader).Read(0xc0001321b0, {0xc00007e000, 0x1000, 0x1000})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0xc0000a0720)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:113 +0x21c
bufio.(*Reader).ReadSlice(0xc0000a0720, 0xb0?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:379 +0x60
bufio.(*Reader).ReadLine(0xc0000a0720)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:408 +0x30
net/textproto.(*Reader).readLineSlice(0xc0001321e0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/textproto/reader.go:56 +0xa8
net/textproto.(*Reader).ReadLine(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/textproto/reader.go:39
net/http.readRequest(0xc0001a6050?)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/request.go:1041 +0xa0
net/http.(*conn).readRequest(0xc00040e1b0, {0x100bfd580, 0xc000020050})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:999 +0x43c
net/http.(*conn).serve(0xc00040e1b0, {0x100bfd548, 0xc0002f2300})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:1934 +0x480
created by net/http.(*Server).Serve in goroutine 22
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3086 +0x63c

goroutine 84 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x10234c960, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc0003064a0, 0xc0004c6078?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000306480, {0xc0004c6078, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000306480, {0xc0004c6078, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc000404000, {0xc0004c6078, 0x8, 0x8})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x8c
github.com/Trinoooo/eggie_kv/storage/server.longConnection(0xc0001836c0, {0xc000388088, 0x8, 0x8})
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:94 +0x240
github.com/Trinoooo/eggie_kv/storage/server.mockClient.func2()
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:44 +0xa0
created by github.com/Trinoooo/eggie_kv/storage/server.mockClient in goroutine 23
	/Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:42 +0x14c

goroutine 521 [IO wait]:
internal/poll.runtime_pollWait(0x129be1808, 0x72)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc000136020, 0xc0000c4071?, 0x0)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000136000, {0xc0000c4071, 0x1, 0x1})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000136000, {0xc0000c4071, 0x1, 0x1})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc0001a6048, {0xc0000c4071, 0x1, 0x1})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x8c
net/http.(*connReader).backgroundRead(0xc0000c4060)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:683 +0x88
created by net/http.(*connReader).startBackgroundRead in goroutine 519
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:679 +0x1d4

额外信息：

  • 我为了方便调试，多路复用API使用了bsd系统的kqueue（因为我用的mac），可能会造成你解决问题时的困扰。
  • 这个问题只有把concurrency调整到18（不同计算机可能不同）以上才会出现，我推测是并发度低问题难以复现。

感谢关注！辛苦有思路随时联系 :)

@codecov Codecov
Copy link

codecov bot commented Jun 19, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 72.81796% with 109 lines in your changes missing coverage. Please review.

Project coverage is 72.81%. Comparing base (48d8cea) to head (6945c83).
Report is 1 commits behind head on main.

Files Patch % Lines
storage/server/processor.go 57.76% 55 Missing and 13 partials ⚠️
storage/server/reactor_server.go 83.74% 21 Missing and 12 partials ⚠️
storage/server/handler.go 68.42% 6 Missing ⚠️
storage/server/metrics.go 88.88% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main      #26      +/-   ##
==========================================
+ Coverage   70.53%   72.81%   +2.28%     
==========================================
  Files           3        4       +1     
  Lines         733      401     -332     
==========================================
- Hits          517      292     -225     
+ Misses        140       83      -57     
+ Partials       76       26      -50

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Trinoooo
Copy link
Owner Author

额外补充一点哈，出现问题的是ReactorServer，其他server实现请忽略，后面我会清理一下

@Trinoooo
Copy link
Owner Author

补充更加详细的上下文信息：

2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:106: reactor #0 register event success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:112: reactor #0 handle event
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:106: reactor #2 register event success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:112: reactor #2 handle event
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:164: waiter #0 sent evt to reactor success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:142: waiter #0 ready to wait event trigger
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50882
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50882
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 114
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:164: waiter #2 sent evt to reactor success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:106: reactor #0 register event success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:142: waiter #2 ready to wait event trigger
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:154: waiter #0 event trigger success, evts: []poller.Pevent{poller.Pevent{ConnFd:0x52, Operation:-1, Flag:21, UserData:(*uint8)(0xc0004300c0)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}, poller.Pevent{ConnFd:0x0, Operation:0, Flag:0, UserData:(*uint8)(nil)}}, n: 1
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 119
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{119 -1 21 0xc0004a6070}] 127.0.0.1:50882 127.0.0.1:9999
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:112: reactor #0 handle event
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:164: waiter #0 sent evt to reactor success
2024/06/20 09:09:56 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:142: waiter #0 ready to wait event trigger
    server_test.go:96: read tcp 127.0.0.1:50882->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #1 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50851
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50851
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 57
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50853
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50853
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{57 -1 21 0xc0001a0db0}] 127.0.0.1:50851 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 64
    server_test.go:96: read tcp 127.0.0.1:50851->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #1 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{64 -1 21 0xc000430060}] 127.0.0.1:50856 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50856
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #1 ready to register event, wrapper fd: 61
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50856
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{61 -1 21 0xc000430030}] 127.0.0.1:50853 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50853->127.0.0.1:9999: read: connection reset by peer
    server_test.go:96: read tcp 127.0.0.1:50856->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50858
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #1 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50858
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #1 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #1 ready to register event, wrapper fd: 70
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 74
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{74 -1 21 0xc00032e210}] 127.0.0.1:50859 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50859
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50859
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{70 -1 21 0xc0001a0dd0}] 127.0.0.1:50858 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50858->127.0.0.1:9999: read: connection reset by peer
    server_test.go:96: read tcp 127.0.0.1:50859->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50857
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50857
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 69
    server_test.go:96: read tcp 127.0.0.1:50857->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{69 -1 21 0xc0001a0dc0}] 127.0.0.1:50857 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50867
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50867
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #1 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50868
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #1 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #1 ready to register event, wrapper fd: 91
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50868
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 90
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{91 -1 21 0xc0004300f0}] 127.0.0.1:50868 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50868->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{90 -1 21 0xc000094310}] 127.0.0.1:50867 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50867->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50865
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #1 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #1 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50865
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #1 ready to register event, wrapper fd: 86
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{86 -1 21 0xc0004300e0}] 127.0.0.1:50865 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50865->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50870
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #1 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50870
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #1 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #1 ready to register event, wrapper fd: 95
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{95 -1 21 0xc000094320}] 127.0.0.1:50870 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50870->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50872
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 107
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50872
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50876
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50874
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50876
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50874
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{107 -1 21 0xc00039a090}] 127.0.0.1:50876 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 99
    server_test.go:96: EOF
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50864
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{99 -1 21 0xc00032e290}] 127.0.0.1:50872 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50864
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
    server_test.go:96: read tcp 127.0.0.1:50872->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50877
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #2 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50877
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #2 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #2 ready to register event, wrapper fd: 104
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 83
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{104 -1 21 0xc0005a6080}] 127.0.0.1:50874 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{83 -1 21 0xc0004300d0}] 127.0.0.1:50864 127.0.0.1:9999
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
    server_test.go:96: read tcp 127.0.0.1:50874->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 109
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
    server_test.go:96: read tcp 127.0.0.1:50864->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{109 -1 21 0xc000430170}] 127.0.0.1:50877 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50877->127.0.0.1:9999: read: connection reset by peer
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:224: dispatcher receive connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:229: dispatcher send connection to reactor #0 success
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:99: [long] client recv response successfully 127.0.0.1:9999 127.0.0.1:50863
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:76: reactor #0 receive output connection
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/server_test.go:92: [long] client send request successfully 127.0.0.1:9999 127.0.0.1:50863
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:88: reactor #0 ready to register event, wrapper fd: 82
2024/06/20 09:09:57 /Users/sujun/go/src/github.com/Trinoooo/eggieKv/storage/server/reactor_server.go:102: bad file descriptor [{82 -1 21 0xc0004300c0}] 127.0.0.1:50863 127.0.0.1:9999
    server_test.go:96: read tcp 127.0.0.1:50863->127.0.0.1:9999: read: connection reset by peer

@Trinoooo
fix(server): 将net.Conn换成自己手搓的conn后，问题解决了，concurrency=30
2012223 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
fix：解了些bug
5f63671 
1. 增加prometheus打点监控，便于排查问题。没有使用在应用程序开个端口prometheus
服务器主动拉取这种模式，而是启动PushGateway作为sidecar中转，主要考虑是调试阶段
程序运行时间短，不好收集指标。
2. 修复runtime: marked free object in span 0x12a642b58, elemsize=32 freeindex=18
(bad use of unsafe.Pointer? try -d=checkptr)问题，原因是*conn通过unsafe.Pointer
转*byte会导致conn结构超过8字节的部分失去引用被gc（个人推测，还得深入研究runtime
包明确结论；至于为什么要用`udata = *(**byte)(unsafe.Pointer(&conn))`是因为
cloudwego/netpoll是这样用的）
3. syscall.INTR可能导致系统调用被中断，给waiter、reactor、acceptor这类遇到syscall
错误会崩溃的协程加特判
4. 给各环节间通信管道加buf，提高整体协作效率。但从accept qps看没有明显优化收益，
后续量化请求qps给个优化结论。
5. 由于accept性能瓶颈（1s获取10个链接）存在，高并发场景下会出现由于backlog满导致
大量请求被server端reset。后续动作考虑看下我的用法是否正确，以及池化链接对象那内存
分配回收性能收益。

后续动作：kqueue改成非阻塞水平触发，reactor负责协议解析，handler纯处理业务逻辑。

Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
feat: kqueue改成水平触发模式；将协议编解码逻辑委托
8aebae5 
给reactor；不使用kqueue的udata存放conn，因为会有奇怪的问题

Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
fix: test config & concurrency number
533cedb 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): 修改currency到一个较低值 & 单测运行器改为macos-latest
d1139e6 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
fix(server): cli对不存在的引用报错
934d1d8 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): lint config & test config
c5b4d97 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): ci中补充prometheus pushgateway环境搭建步骤
7a7c71c 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): ci中补充prometheus pushgateway环境搭建步骤
98c851e 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): ci中补充prometheus pushgateway环境搭建步骤
2616b42 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
feat(server): 日志增加颜色；测试ci改为包含所有测试用例；修复一些问题
86636a0 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): resolve golang-cilint
8b7912d 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): change test package
0f67d63 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo
chore(server): wrap errors
6945c83 
Signed-off-by: Trino <sujun.trinoooo@gmail.com>
@Trinoooo Trinoooo merged commit 3422fec into main Jun 28, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RocooHash RocooHash RocooHash approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Trinoooo @RocooHash