Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] govomi keepalive handler panic when login with token then login with basic auth #3206

Closed
lubronzhan opened this issue Aug 12, 2023 · 4 comments
Closed

[BUG] govomi keepalive handler panic when login with token then login with basic auth #3206

lubronzhan opened this issue Aug 12, 2023 · 4 comments
Labels
kind/bug lifecycle/stale

Comments

@lubronzhan
Copy link
Contributor

Describe the bug
A clear and concise description of what the bug is.

Goroutine panic when doing local testing with vcsim.
The test will first create a VC session using basic auth, then it will create a VC session with saml token.
Then govmomi keepalive handler will panic

https://github.com/vmware/govmomi/blob/main/session/keepalive/handler.go#L100-L133

To Reproduce
Steps to reproduce the behavior:

  1. create a govmomi.Client and login with basic auth
  2. create another govmomi.Client and login with saml token

Expected behavior
session created with different auth func should coexsit

Affected version
Please provide details on the version used, e.g. release tag, commit, module version, etc.
0.30.7
Screenshots/Debug Output
If applicable, add screenshots or debug output to help explain your problem.

panic: test timed out after 30s
running tests:
	TestGetSession (30s)

goroutine 7133 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2241 +0x328
created by time.goFunc
	/usr/local/go/src/time/sleep.go:176 +0x38

goroutine 1 [chan receive]:
testing.(*T).Run(0x140005029c0, {0x1041f1ca4?, 0x443977feddeb7?}, 0x104b152c8)
	/usr/local/go/src/testing/testing.go:1630 +0x37c
testing.runTests.func1(0x140005029c0?)
	/usr/local/go/src/testing/testing.go:2036 +0x48
testing.tRunner(0x140005029c0, 0x14000805c68)
	/usr/local/go/src/testing/testing.go:1576 +0x10c
testing.runTests(0x14000476500?, {0x105b2fcc0, 0x2, 0x2}, {0x14000154f40?, 0x40?, 0x105babb40?})
	/usr/local/go/src/testing/testing.go:2034 +0x40c
testing.(*M).Run(0x14000476500)
	/usr/local/go/src/testing/testing.go:1906 +0x58c
main.main()
	_testmain.go:49 +0x1d0

goroutine 52 [runnable]:
syscall.readlen(0x1b?, 0x14000614e88?, 0x8?)
	/usr/local/go/src/syscall/syscall_darwin.go:241 +0xe8
syscall.forkExec({0x1400005e00e?, 0x0?}, {0x14000a54320, 0x2, 0x2}, 0x1047dd440?)
	/usr/local/go/src/syscall/exec_unix.go:221 +0x398
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:255
os.startProcess({0x1400005e00e, 0x6a}, {0x14000a54320, 0x2, 0x2}, 0x14000615188)
	/usr/local/go/src/os/exec_posix.go:54 +0x25c
os.StartProcess({0x1400005e00e, 0x6a}, {0x14000a54320, 0x2, 0x2}, 0x102efcb50?)
	/usr/local/go/src/os/exec.go:109 +0x50
os/exec.(*Cmd).Start(0x1400065ab00)
	/usr/local/go/src/os/exec/exec.go:693 +0x4c4
os/exec.(*Cmd).Run(0x14000c5f1d0?)
	/usr/local/go/src/os/exec/exec.go:587 +0x20
sigs.k8s.io/cluster-api-provider-vsphere/test/helpers/vcsim.Simulator.Run({0x14000471900?, 0x140007a6b70?}, {0x1041e9bb0, 0xa}, {0x14000615300, 0x1, 0x1})
	/Users/xiaoranzhan/work/src/github.com/lubronzhan/cluster-api-provider-vsphere/test/helpers/vcsim/simulator.go:50 +0x124
sigs.k8s.io/cluster-api-provider-vsphere/pkg/session.assertSessionCountEqualTo.func1()
	/Users/xiaoranzhan/work/src/github.com/lubronzhan/cluster-api-provider-vsphere/pkg/session/session_test.go:122 +0xac
reflect.Value.call({0x1046e0040?, 0x14000790e20?, 0x1?}, {0x1041dff26, 0x4}, {0x105be2048, 0x0, 0x3?})
	/usr/local/go/src/reflect/value.go:586 +0x87c
reflect.Value.Call({0x1046e0040?, 0x14000790e20?, 0xffffffffffffffff?}, {0x105be2048?, 0x1?, 0x14000066500?})
	/usr/local/go/src/reflect/value.go:370 +0x90
github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
	/Users/xiaoranzhan/go/pkg/mod/github.com/onsi/gomega@v1.27.10/internal/async_assertion.go:325 +0xf0
github.com/onsi/gomega/internal.(*AsyncAssertion).match(0x1400045f570, {0x104b285f0?, 0x105be2048}, 0x1, {0x0, 0x0, 0x0})
	/Users/xiaoranzhan/go/pkg/mod/github.com/onsi/gomega@v1.27.10/internal/async_assertion.go:540 +0x6c0
github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0x1400045f570, {0x104b285f0, 0x105be2048}, {0x0, 0x0, 0x0})
	/Users/xiaoranzhan/go/pkg/mod/github.com/onsi/gomega@v1.27.10/internal/async_assertion.go:145 +0x7c
sigs.k8s.io/cluster-api-provider-vsphere/pkg/session.assertSessionCountEqualTo(0x1400014b380, 0x14000117ad0, 0x1)
	/Users/xiaoranzhan/work/src/github.com/lubronzhan/cluster-api-provider-vsphere/pkg/session/session_test.go:126 +0xec
sigs.k8s.io/cluster-api-provider-vsphere/pkg/session.TestGetSession(0x14000502b60)
	/Users/xiaoranzhan/work/src/github.com/lubronzhan/cluster-api-provider-vsphere/pkg/session/session_test.go:66 +0x5ac
testing.tRunner(0x14000502b60, 0x104b152c8)
	/usr/local/go/src/testing/testing.go:1576 +0x10c
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1629 +0x368

goroutine 101 [IO wait]:
internal/poll.runtime_pollWait(0x12dad3400, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x140007aa000?, 0x0?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x140007aa000)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x250
net.(*netFD).accept(0x140007aa000)
	/usr/local/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x140009bf440)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x28
net.(*TCPListener).Accept(0x140009bf440)
	/usr/local/go/src/net/tcpsock.go:297 +0x2c
crypto/tls.(*listener).Accept(0x140009bfa40)
	/usr/local/go/src/crypto/tls/tls.go:66 +0x30
net/http.(*Server).Serve(0x140006e2960, {0x104b28200, 0x140009bfa40})
	/usr/local/go/src/net/http/server.go:3059 +0x304
github.com/vmware/govmomi/simulator/internal.(*Server).goServe.func1()
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/simulator/internal/server.go:278 +0x60
created by github.com/vmware/govmomi/simulator/internal.(*Server).goServe
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/simulator/internal/server.go:276 +0x74

goroutine 106 [select]:
github.com/vmware/govmomi/session/keepalive.(*handler).Start.func1()
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:118 +0x7c
created by github.com/vmware/govmomi/session/keepalive.(*handler).Start
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:116 +0x158

goroutine 103 [IO wait]:
internal/poll.runtime_pollWait(0x12dad3220, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x140007aa500?, 0x1400027b800?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140007aa500, {0x1400027b800, 0x800, 0x800})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x140007aa500, {0x1400027b800?, 0x14000ca5598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000186990, {0x1400027b800?, 0x47b?, 0x140007be000?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x140009698a8, {0x1400027b800?, 0x140009698a8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x140004e5e90, {0x104b180c8, 0x140009698a8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x140004e5c00, {0x12dad5798?, 0x14000186990}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x140004e5c00, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x140004e5c00, {0x14000982000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x1400090a0c0, {0x14000982000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x140009b0600)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140009b0600, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x1400079d9e0, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 104 [IO wait]:
internal/poll.runtime_pollWait(0x12dad3310, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x140007aa480?, 0x140009cc000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140007aa480, {0x140009cc000, 0x1300, 0x1300})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x140007aa480, {0x140009cc000?, 0x14000ca6878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000186988, {0x140009cc000?, 0x0?, 0x0?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000aa6cc0, {0x140009cc000?, 0x14000aa6cc0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x140004e5b10, {0x104b180c8, 0x14000aa6cc0})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x140004e5880, {0x12dad5798?, 0x14000186988}, 0x7fffffffffffffff?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x140004e5880, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x140004e5880, {0x140007b7000, 0x1000, 0x140008e5630?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x1400024be60, {0x140007b7000?, 0x102eb1300?, 0x1400010d980?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x140004758c0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140004758c0, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x1400024be60)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 105 [select]:
net/http.(*persistConn).writeLoop(0x1400024be60)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 163 [select]:
github.com/vmware/govmomi/session/keepalive.(*handler).Start.func1()
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:118 +0x7c
created by github.com/vmware/govmomi/session/keepalive.(*handler).Start
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:116 +0x158

goroutine 46 [select]:
github.com/vmware/govmomi/session/keepalive.(*handler).Start.func1()
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:118 +0x7c
created by github.com/vmware/govmomi/session/keepalive.(*handler).Start
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:116 +0x158

goroutine 136 [select]:
net/http.(*persistConn).writeLoop(0x14000414900)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 48 [IO wait]:
internal/poll.runtime_pollWait(0x12dad3040, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098c200?, 0x140009f4000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098c200, {0x140009f4000, 0x240, 0x240})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098c200, {0x140009f4000?, 0x14000565598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000130288, {0x140009f4000?, 0x4fb?, 0x140009e6340?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000aa6ab0, {0x140009f4000?, 0x14000aa6ab0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000190290, {0x104b180c8, 0x14000aa6ab0})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000190000, {0x12dad5798?, 0x14000130288}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000190000, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000190000, {0x140009e5000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x1400090b350, {0x140009e5000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000990ba0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000990ba0, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x140009846c0, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 84 [IO wait]:
internal/poll.runtime_pollWait(0x12dad3130, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098c180?, 0x14000b82700?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098c180, {0x14000b82700, 0x700, 0x700})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098c180, {0x14000b82700?, 0x14000576878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400000e8c8, {0x14000b82700?, 0x140005767c8?, 0x102eb57f8?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x140009689a8, {0x14000b82700?, 0x140009689a8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000a08290, {0x104b180c8, 0x140009689a8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000a08000, {0x12dad5798?, 0x1400000e8c8}, 0x102ec0b40?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000a08000, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000a08000, {0x14000b94000, 0x1000, 0x140007f93b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000248ea0, {0x14000b94000?, 0x102eb1300?, 0x1400005d260?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x14000b8a7e0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000b8a7e0, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000248ea0)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 85 [select]:
net/http.(*persistConn).writeLoop(0x14000248ea0)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 129 [select]:
github.com/vmware/govmomi/session/keepalive.(*handler).Start.func1()
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:118 +0x7c
created by github.com/vmware/govmomi/session/keepalive.(*handler).Start
	/Users/xiaoranzhan/go/pkg/mod/github.com/vmware/govmomi@v0.30.7/session/keepalive/handler.go:116 +0x158

goroutine 135 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2e60, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000b0a300?, 0x14000d96000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000b0a300, {0x14000d96000, 0x1300, 0x1300})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000b0a300, {0x14000d96000?, 0x14000573878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000130450, {0x14000d96000?, 0x0?, 0x0?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000ce28, {0x14000d96000?, 0x1400000ce28?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000191b10, {0x104b180c8, 0x1400000ce28})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000191880, {0x12dad5798?, 0x14000130450}, 0x7fffffffffffffff?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000191880, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000191880, {0x14000d26000, 0x1000, 0x14000b867d0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000414900, {0x14000d26000?, 0x102eb1300?, 0x1400098a840?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x14000d214a0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000d214a0, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000414900)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 138 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2c80, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098cb80?, 0x14000d3c000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098cb80, {0x14000d3c000, 0x500, 0x500})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098cb80, {0x14000d3c000?, 0x14000551598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140001304c8, {0x14000d3c000?, 0xa7b?, 0x140009e69c0?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000488438, {0x14000d3c000?, 0x14000488438?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000191e90, {0x104b180c8, 0x14000488438})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000191c00, {0x12dad5798?, 0x140001304c8}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000191c00, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000191c00, {0x14000d10000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x14000d1a9f0, {0x14000d10000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000c4bbc0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000c4bbc0, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x14000c18b40, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 118 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2f50, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000b0a380?, 0x14000e74000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000b0a380, {0x14000e74000, 0x2a80, 0x2a80})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000b0a380, {0x14000e74000?, 0x140005cf598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000496868, {0x14000e74000?, 0x12fb?, 0x1400018b040?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x140009bfae8, {0x14000e74000?, 0x140009bfae8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000686990, {0x104b180c8, 0x140009bfae8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000686700, {0x12dad5798?, 0x14000496868}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000686700, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000686700, {0x14000b11000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x14000ac50b0, {0x14000b11000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000a87a40)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000a87a40, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x140008dd830, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 121 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2d70, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098cb00?, 0x14000da5000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098cb00, {0x14000da5000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098cb00, {0x14000da5000?, 0x14000d92878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000496908, {0x14000da5000?, 0x14000d927f8?, 0x102fb83ec?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000aa7c68, {0x14000da5000?, 0x14000aa7c68?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000686d10, {0x104b180c8, 0x14000aa7c68})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000686a80, {0x12dad5798?, 0x14000496908}, 0x7fffffffffffffff?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000686a80, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000686a80, {0x14000b21000, 0x1000, 0x14000b026d0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000393b00, {0x14000b21000?, 0x102eb1300?, 0x14000197f20?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x14000b61a40)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000b61a40, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000393b00)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 122 [select]:
net/http.(*persistConn).writeLoop(0x14000393b00)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 141 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2b90, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000d88600?, 0x14000e24000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000d88600, {0x14000e24000, 0x2000, 0x2000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000d88600, {0x14000e24000?, 0x14000d93878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000130588, {0x14000e24000?, 0x14000d937a8?, 0x102eee9e4?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000c7c8, {0x14000e24000?, 0x1400000c7c8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000d46290, {0x104b180c8, 0x1400000c7c8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000d46000, {0x12dad5798?, 0x14000130588}, 0x700?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000d46000, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000d46000, {0x14000d5c000, 0x1000, 0x14000b343e0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000db2120, {0x14000d5c000?, 0x102eb1300?, 0x1400098ae40?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x14000d5a720)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000d5a720, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000db2120)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 91 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2aa0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000d88680?, 0x14000d97300?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000d88680, {0x14000d97300, 0x1300, 0x1300})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000d88680, {0x14000d97300?, 0x14000c41598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400000ec08, {0x14000d97300?, 0x12fb?, 0x14000583d40?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000aa7fc8, {0x14000d97300?, 0x14000aa7fc8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000a08610, {0x104b180c8, 0x14000aa7fc8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000a08380, {0x12dad5798?, 0x1400000ec08}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000a08380, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000a08380, {0x14000dcc000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x14000d9f110, {0x14000dcc000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000dca7e0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000dca7e0, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x14000d81290, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 142 [select]:
net/http.(*persistConn).writeLoop(0x14000db2120)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 146 [IO wait]:
internal/poll.runtime_pollWait(0x12dad29b0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000b0a200?, 0x14000598000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000b0a200, {0x14000598000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000b0a200, {0x14000598000?, 0x14000d8e878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000186210, {0x14000598000?, 0x14000d8e7f8?, 0x102fb83ec?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x140009be3d8, {0x14000598000?, 0x140009be3d8?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000d46610, {0x104b180c8, 0x140009be3d8})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000d46380, {0x12dad5798?, 0x14000186210}, 0x7fffffffffffffff?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000d46380, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000d46380, {0x14000b1d000, 0x1000, 0x14000b58c70?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x140007da360, {0x14000b1d000?, 0x102eb1300?, 0x1400098a480?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x140000a0780)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140000a0780, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x140007da360)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 125 [IO wait]:
internal/poll.runtime_pollWait(0x12dad28c0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x14000b0a280?, 0x14000172000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000b0a280, {0x14000172000, 0x600, 0x600})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x14000b0a280, {0x14000172000?, 0x14000565598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140004960a8, {0x14000172000?, 0x197b?, 0x140009e6ea0?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000cba0, {0x14000172000?, 0x1400000cba0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000687410, {0x104b180c8, 0x1400000cba0})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000687180, {0x12dad5798?, 0x140004960a8}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000687180, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000687180, {0x14000b7f000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x14000e554d0, {0x14000b7f000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000e14cc0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000e14cc0, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x14000e127e0, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 147 [select]:
net/http.(*persistConn).writeLoop(0x140007da360)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 128 [IO wait]:
internal/poll.runtime_pollWait(0x12dad27d0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x140007aa800?, 0x14000770000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140007aa800, {0x14000770000, 0x2000, 0x2000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x140007aa800, {0x14000770000?, 0x14000cad878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000496110, {0x14000770000?, 0x14000cad7a8?, 0x102eee9e4?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x14000489638, {0x14000770000?, 0x14000489638?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000687b10, {0x104b180c8, 0x14000489638})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000687880, {0x12dad5798?, 0x14000496110}, 0x102ec0b40?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000687880, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000687880, {0x140004f4000, 0x1000, 0x140007df0a0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000d36b40, {0x140004f4000?, 0x102eb1300?, 0x1400010c540?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x140004eb020)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140004eb020, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000d36b40)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 149 [IO wait]:
internal/poll.runtime_pollWait(0x12dad26e0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x140007aa880?, 0x140009cd300?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140007aa880, {0x140009cd300, 0x1300, 0x1300})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x140007aa880, {0x140009cd300?, 0x1400060f598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000186380, {0x140009cd300?, 0x12fb?, 0x1400018ba00?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x140009be6f0, {0x140009cd300?, 0x140009be6f0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000d46990, {0x104b180c8, 0x140009be6f0})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000d46700, {0x12dad5798?, 0x14000186380}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000d46700, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000d46700, {0x140005b1000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x140007a7fb0, {0x140005b1000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x140000a1680)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140000a1680, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x1400079cc60, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 161 [select]:
net/http.(*persistConn).writeLoop(0x14000d36b40)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214

goroutine 151 [IO wait]:
internal/poll.runtime_pollWait(0x12dad25f0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098c580?, 0x140006afc00?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098c580, {0x140006afc00, 0x700, 0x700})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098c580, {0x140006afc00?, 0x14000d8f878?, 0x10314027c?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000186398, {0x140006afc00?, 0x14000d8f7a8?, 0x102eee9e4?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000cd38, {0x140006afc00?, 0x1400000cd38?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000d46d10, {0x104b180c8, 0x1400000cd38})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000d46a80, {0x12dad5798?, 0x14000186398}, 0x7fffffffffffffff?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000d46a80, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000d46a80, {0x140005b5000, 0x1000, 0x14000b032c0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*persistConn).Read(0x14000e307e0, {0x140005b5000?, 0x102eb1300?, 0x1400098b140?})
	/usr/local/go/src/net/http/transport.go:1943 +0x50
bufio.(*Reader).fill(0x140008e3da0)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x140008e3da0, 0x1)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*persistConn).readLoop(0x14000e307e0)
	/usr/local/go/src/net/http/transport.go:2107 +0x144
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1765 +0x11c8

goroutine 108 [IO wait]:
internal/poll.runtime_pollWait(0x12dad2500, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0xa0
internal/poll.(*pollDesc).wait(0x1400098c600?, 0x140007c1000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x1400098c600, {0x140007c1000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x200
net.(*netFD).Read(0x1400098c600, {0x140007c1000?, 0x14000561598?, 0x102eee9e4?})
	/usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000130340, {0x140007c1000?, 0x197b?, 0x140007be680?})
	/usr/local/go/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000ccf0, {0x140007c1000?, 0x1400000ccf0?, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:788 +0x40
bytes.(*Buffer).ReadFrom(0x14000a08990, {0x104b180c8, 0x1400000ccf0})
	/usr/local/go/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x14000a08700, {0x12dad5798?, 0x14000130340}, 0x102f56a0c?)
	/usr/local/go/src/crypto/tls/conn.go:810 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x14000a08700, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:617 +0xd8
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0x14000a08700, {0x140007b1000, 0x1000, 0x140007a70b0?})
	/usr/local/go/src/crypto/tls/conn.go:1316 +0x178
net/http.(*connReader).Read(0x14000d1bcb0, {0x140007b1000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:782 +0x234
bufio.(*Reader).fill(0x14000d5af60)
	/usr/local/go/src/bufio/bufio.go:106 +0xfc
bufio.(*Reader).Peek(0x14000d5af60, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x60
net/http.(*conn).serve(0x14000c187e0, {0x104b35be0, 0x1400056da10})
	/usr/local/go/src/net/http/server.go:2030 +0x694
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x520

goroutine 152 [select]:
net/http.(*persistConn).writeLoop(0x14000e307e0)
	/usr/local/go/src/net/http/transport.go:2410 +0x9c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1766 +0x1214
FAIL	sigs.k8s.io/cluster-api-provider-vsphere/pkg/session	30.384s
FAIL

Additional context
Add any other context about the problem here.
@dougm
Copy link
Member

dougm commented Aug 24, 2023

@lubronzhan can you share a test that reproduces this?

@lubronzhan
Copy link
Contributor Author

Hi @dougm I was adding saml support to CAPV. below are the link to the tests
kubernetes-sigs/cluster-api-provider-vsphere@9347369 (#2171)

I just added loginbytoken func to GetOrCreate function https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/2171/files#diff-59fbb0bb5dfeeb4f123f1e9e0a3d12c38a60b839e113b29868cb905755639e13R130

It should login with bear token if userInfo is present, or login with saml sso if cert key pair are presented.

If you want to run the spefific test, you can set GOVC_BIN_PATH then run the test

go test -timeout 30s -run ^TestGetSession$ sigs.k8s.io/cluster-api-provider-vsphere/pkg/session

Thank you

@dougm
Copy link
Member

dougm commented Aug 25, 2023

I tried your branch, I see a timeout failure, but not panic:

=== RUN   TestGetSession
    session_test.go:126: 
        Timed out after 30.015s.
        Expected
            <bool>: false
        to be true
--- FAIL: TestGetSession (30.64s)
FAIL
FAIL	sigs.k8s.io/cluster-api-provider-vsphere/pkg/session	31.183s
FAIL

The branch diff and capv layers are a bit much to go through, can you reproduce with just govmomi? Such as by changing one of these tests?

govmomi/session/keepalive/handler_test.go

Line 56 in 72d86e9

sc := soap.NewClient(c.URL(), true)

Or:

govmomi/session/keepalive/example_test.go

Line 42 in 72d86e9

func ExampleHandlerSOAP() {

@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Mark as fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug lifecycle/stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dougm @tenthirtyam @lubronzhan