Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: net.runtime_pollWait Error of concurrent http post #16079

Closed
woria opened this issue Jun 16, 2016 · 9 comments

Comments

Projects
None yet
5 participants
@woria
Copy link

commented Jun 16, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    1.62
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOBIN=""
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOOS="linux"
    GOPATH="/mnt/windows/web_application"
    GORACE=""
    GOROOT="/apps/lib/go"
    GOTOOLDIR="/apps/lib/go/pkg/tool/linux_amd64"
    GO15VENDOREXPERIMENT="1"
    CC="gcc"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
    CXX="g++"
    CGO_ENABLED="1"
  3. What did you do?
    code like this:
func (this *instance) pushMultiMsg(pushList) {
        ......
        for {
            //get userId from redis list
            userId := this.model.PopList(pushList)

            if userId == "" {
                // task finish
                break
            }
            ......
            go this.concurrentPush(userId)
        }
        ......
    }

func (this *instance) concurrentPush(userId) {
    ......
    client := &http.Client{}
    req, err := http.NewRequest("POST", URL, bytes.NewReader(data))
    req.Close = true
    req.Header.Set("Content-Type", "application/json; charset=utf-8")
    resp, err := client.Do(req)
    defer resp.Body.Close()
    ......
}

a) get userId from redis list [500000 msg data in list]

b) use a new goroutine for post data to URL [total 10000 goroutine]

c) sometimes post all successfully , and sometimes a small number of list post successfully

ERROR LOG:

goroutine 13367937 [IO wait]:
net.runtime_pollWait(0x2b20f404a278, 0x72, 0xc82644c400)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820a3c0d0, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820a3c0d0, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820a3c070, 0xc82644c400, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8209b63d0, 0xc82644c400, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820573cb0, 0x2b20e88c8b80, 0xc8209b63d0, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc822564600, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc822564600, 0xc8212f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
goroutine 13367250 [select]:
net/http.(*persistConn).writeLoop(0xc82a25c820)
        /apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368465 [IO wait]:
net.runtime_pollWait(0x2b20f4ac0f78, 0x72, 0xc8254af000)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc821bf8370, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc821bf8370, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc821bf8310, 0xc8254af000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8204e4080, 0xc8254af000, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc821379080, 0x2b20e88c8b80, 0xc8204e4080, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8206d6300, 0x16, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).readHandshake(0xc8206d6300, 0x0, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:800 +0xd3
--
goroutine 13368555 [select]:
net/http.(*persistConn).writeLoop(0xc82a282c30)
        /apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368135 [IO wait]:
net.runtime_pollWait(0x2b20f738e428, 0x72, 0xc821f71c00)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8205d2b50, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8205d2b50, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8205d2af0, 0xc821f71c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820158218, 0xc821f71c00, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc828b11fb0, 0x2b20e88c8b80, 0xc820158218, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc82af6a600, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82af6a600, 0xc8258a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435
msgpush/lib/push.(*instance).concurrentPusher(0xc82aaf9bd0, 0xc827454300, 0x210, 0x2e5, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc82b31bfe0, 0x1c, ...)
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(*instance).pushMultiMsg
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368994 [IO wait]:
net.runtime_pollWait(0x2b20f7ec79a0, 0x72, 0xc824130000)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82ac20df0, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82ac20df0, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82ac20d90, 0xc824130000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc823ab4120, 0xc824130000, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc824acc570, 0x2b20e88c8b80, 0xc823ab4120, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8206d6900, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8206d6900, 0xc822100000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435
msgpush/lib/push.(*instance).concurrentPusher(0xc8201400f0, 0xc820709d40, 0x203, 0x211, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc8205d5280, 0x1c, ...)
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(*instance).pushMultiMsg
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368565 [IO wait]:
net.runtime_pollWait(0x2b20f7e14700, 0x72, 0xc820299000)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8245a8370, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8245a8370, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8245a8310, 0xc820299000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc821f00018, 0xc820299000, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820677cb0, 0x2b20e88c8b80, 0xc821f00018, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc82272f800, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82272f800, 0xc824e4a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /apps/lib/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82a2f0820)
        /apps/lib/go/src/net/http/transport.go:1073 +0x177
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:857 +0x10a6

goroutine 13368225 [IO wait]:
net.runtime_pollWait(0x2b20ebb4b4c0, 0x72, 0xc82017d800)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820157090, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820157090, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820157030, 0xc82017d800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8205603e8, 0xc82017d800, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8205074d0, 0x2b20e88c8b80, 0xc8205603e8, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc825894f00, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc825894f00, 0xc820928000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
goroutine 13367135 [select]:
net/http.(*persistConn).writeLoop(0xc82090b1e0)
        /apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368463 [IO wait]:
net.runtime_pollWait(0x2b20f7e8df80, 0x72, 0xc82af04c00)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82ac9c450, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82ac9c450, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82ac9c3f0, 0xc82af04c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820040300, 0xc82af04c00, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820498a80, 0x2b20e88c8b80, 0xc820040300, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc829d2e900, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc829d2e900, 0xc821c14000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435
msgpush/lib/push.(*instance).concurrentPusher(0xc82aaf9bd0, 0xc820beec00, 0x210, 0x2e5, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc8220e5dc0, 0x1c, ...)
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(*instance).pushMultiMsg
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13366233 [IO wait]:
net.runtime_pollWait(0x2b20f7e586f8, 0x72, 0xc820bf5800)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82339a1b0, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82339a1b0, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82339a150, 0xc820bf5800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82111a408, 0xc820bf5800, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8243d6cf0, 0x2b20e88c8b80, 0xc82111a408, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc821dddb00, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc821dddb00, 0xc82aefd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
goroutine 13368168 [select]:
net/http.(*persistConn).writeLoop(0xc820169930)
        /apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368659 [IO wait]:
net.runtime_pollWait(0x2b20f7f4dc88, 0x72, 0xc8219b3400)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8238033a0, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8238033a0, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc823803340, 0xc8219b3400, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820a1a028, 0xc8219b3400, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82af54900, 0x2b20e88c8b80, 0xc820a1a028, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820295200, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820295200, 0xc821a1c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /apps/lib/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82029a9c0)
        /apps/lib/go/src/net/http/transport.go:1073 +0x177
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:857 +0x10a6

goroutine 13368299 [IO wait]:
net.runtime_pollWait(0x2b20f89fe5c0, 0x72, 0xc824a5a800)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82a5cd020, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82a5cd020, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82a5ccfc0, 0xc824a5a800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820040160, 0xc824a5a800, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8213a6480, 0x2b20e88c8b80, 0xc820040160, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc82b772f00, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82b772f00, 0xc820df0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435
msgpush/lib/push.(*instance).concurrentPusher(0xc8201400f0, 0xc820876240, 0x203, 0x211, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc823303960, 0x1c, ...)
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(*instance).pushMultiMsg
        /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368425 [IO wait]:
net.runtime_pollWait(0x2b20f73da270, 0x72, 0xc820ea2c00)
        /apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82b598b50, 0x72, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82b598b50, 0x0, 0x0)
        /apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82b598af0, 0xc820ea2c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
        /apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82a2a8028, 0xc820ea2c00, 0x400, 0x400, 0x0, 0x0, 0x0)
        /apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820511410, 0x2b20e88c8b80, 0xc82a2a8028, 0x5, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820a99800, 0x9b4117, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820a99800, 0xc8224e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /apps/lib/go/src/crypto/tls/conn.go:939 +0x167
--
goroutine 13367415 [select]:
net/http.(*persistConn).writeLoop(0xc8202a0750)
        /apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(*Transport).dialConn
        /apps/lib/go/src/net/http/transport.go:858 +0x10cb
  1. What did you expect to see?
    http post data successfully
  2. What did you see instead?
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2016

It is likely your program has a data race. Please post a complete code sample the demonstrates the problem.

On 16 Jun 2016, at 18:23, woria notifications@github.com wrote:

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
1.62

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/mnt/windows/web_application"
GORACE=""
GOROOT="/apps/lib/go"
GOTOOLDIR="/apps/lib/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?
code like this:

func (this *instance) pushMultiMsg(pushList) {
......
for {
//get userId from redis list
userId := this.model.PopList(pushList)

        if userId == "" {
            // task finish
            break
        }
        ......
        go this.concurrentPush(userId)
    }
    ......
}

func (this *instance) concurrentPush(userId) {
......
client := &http.Client{}
req, err := http.NewRequest("POST", URL, bytes.NewReader(data))
req.Close = true
req.Header.Set("Content-Type", "application/json; charset=utf-8")
resp, err := client.Do(req)
defer resp.Body.Close()
......
}
a) get userId from redis list [500000 msg data in list]

b) use a new goroutine for post data to URL [total 10000 goroutine]

c) sometimes post all successfully , and sometimes a small number of list post successfully

ERROR LOG:

goroutine 13367937 [IO wait]:
net.runtime_pollWait(0x2b20f404a278, 0x72, 0xc82644c400)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc820a3c0d0, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc820a3c0d0, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc820a3c070, 0xc82644c400, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc8209b63d0, 0xc82644c400, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc820573cb0, 0x2b20e88c8b80, 0xc8209b63d0, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc822564600, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc822564600, 0xc8212f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

goroutine 13367250 [select]:
net/http.(_persistConn).writeLoop(0xc82a25c820)
/apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368465 [IO wait]:
net.runtime_pollWait(0x2b20f4ac0f78, 0x72, 0xc8254af000)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc821bf8370, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc821bf8370, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc821bf8310, 0xc8254af000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc8204e4080, 0xc8254af000, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc821379080, 0x2b20e88c8b80, 0xc8204e4080, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc8206d6300, 0x16, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).readHandshake(0xc8206d6300, 0x0, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:800 +0xd3

goroutine 13368555 [select]:
net/http.(_persistConn).writeLoop(0xc82a282c30)
/apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368135 [IO wait]:
net.runtime_pollWait(0x2b20f738e428, 0x72, 0xc821f71c00)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc8205d2b50, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc8205d2b50, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc8205d2af0, 0xc821f71c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc820158218, 0xc821f71c00, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc828b11fb0, 0x2b20e88c8b80, 0xc820158218, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc82af6a600, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82af6a600, 0xc8258a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435

msgpush/lib/push.(_instance).concurrentPusher(0xc82aaf9bd0, 0xc827454300, 0x210, 0x2e5, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc82b31bfe0, 0x1c, ...)
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(_instance).pushMultiMsg
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368994 [IO wait]:
net.runtime_pollWait(0x2b20f7ec79a0, 0x72, 0xc824130000)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82ac20df0, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82ac20df0, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82ac20d90, 0xc824130000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc823ab4120, 0xc824130000, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc824acc570, 0x2b20e88c8b80, 0xc823ab4120, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc8206d6900, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8206d6900, 0xc822100000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435

msgpush/lib/push.(_instance).concurrentPusher(0xc8201400f0, 0xc820709d40, 0x203, 0x211, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc8205d5280, 0x1c, ...)
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(_instance).pushMultiMsg
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368565 [IO wait]:
net.runtime_pollWait(0x2b20f7e14700, 0x72, 0xc820299000)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc8245a8370, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc8245a8370, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc8245a8310, 0xc820299000, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc821f00018, 0xc820299000, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc820677cb0, 0x2b20e88c8b80, 0xc821f00018, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc82272f800, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82272f800, 0xc824e4a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /apps/lib/go/src/bufio/bufio.go:132 +0xcc

net/http.(_persistConn).readLoop(0xc82a2f0820)
/apps/lib/go/src/net/http/transport.go:1073 +0x177
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:857 +0x10a6

goroutine 13368225 [IO wait]:
net.runtime_pollWait(0x2b20ebb4b4c0, 0x72, 0xc82017d800)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc820157090, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc820157090, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc820157030, 0xc82017d800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc8205603e8, 0xc82017d800, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc8205074d0, 0x2b20e88c8b80, 0xc8205603e8, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc825894f00, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc825894f00, 0xc820928000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

goroutine 13367135 [select]:
net/http.(_persistConn).writeLoop(0xc82090b1e0)
/apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368463 [IO wait]:
net.runtime_pollWait(0x2b20f7e8df80, 0x72, 0xc82af04c00)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82ac9c450, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82ac9c450, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82ac9c3f0, 0xc82af04c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc820040300, 0xc82af04c00, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc820498a80, 0x2b20e88c8b80, 0xc820040300, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc829d2e900, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc829d2e900, 0xc821c14000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435

msgpush/lib/push.(_instance).concurrentPusher(0xc82aaf9bd0, 0xc820beec00, 0x210, 0x2e5, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc8220e5dc0, 0x1c, ...)
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(_instance).pushMultiMsg
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13366233 [IO wait]:
net.runtime_pollWait(0x2b20f7e586f8, 0x72, 0xc820bf5800)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82339a1b0, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82339a1b0, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82339a150, 0xc820bf5800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc82111a408, 0xc820bf5800, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc8243d6cf0, 0x2b20e88c8b80, 0xc82111a408, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc821dddb00, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc821dddb00, 0xc82aefd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

goroutine 13368168 [select]:
net/http.(_persistConn).writeLoop(0xc820169930)
/apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:858 +0x10cb

goroutine 13368659 [IO wait]:
net.runtime_pollWait(0x2b20f7f4dc88, 0x72, 0xc8219b3400)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc8238033a0, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc8238033a0, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc823803340, 0xc8219b3400, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc820a1a028, 0xc8219b3400, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc82af54900, 0x2b20e88c8b80, 0xc820a1a028, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc820295200, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820295200, 0xc821a1c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /apps/lib/go/src/bufio/bufio.go:132 +0xcc

net/http.(_persistConn).readLoop(0xc82029a9c0)
/apps/lib/go/src/net/http/transport.go:1073 +0x177
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:857 +0x10a6

goroutine 13368299 [IO wait]:
net.runtime_pollWait(0x2b20f89fe5c0, 0x72, 0xc824a5a800)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82a5cd020, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82a5cd020, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82a5ccfc0, 0xc824a5a800, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc820040160, 0xc824a5a800, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc8213a6480, 0x2b20e88c8b80, 0xc820040160, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc82b772f00, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc82b772f00, 0xc820df0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

    /mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:427 +0x435

msgpush/lib/push.(_instance).concurrentPusher(0xc8201400f0, 0xc820876240, 0x203, 0x211, 0x98c6e0, 0x47, 0xc820116360, 0x12, 0xc823303960, 0x1c, ...)
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:385 +0x7c2
created by msgpush/lib/push.(_instance).pushMultiMsg
/mnt/windows/web_application/msg-push/src/msgpush/lib/push/push.go:302 +0x97a

goroutine 13368425 [IO wait]:
net.runtime_pollWait(0x2b20f73da270, 0x72, 0xc820ea2c00)
/apps/lib/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82b598b50, 0x72, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82b598b50, 0x0, 0x0)
/apps/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82b598af0, 0xc820ea2c00, 0x400, 0x400, 0x0, 0x2b20e88c3050, 0xc82000a1a0)
/apps/lib/go/src/net/fd_unix.go:250 +0x23a
net.(_conn).Read(0xc82a2a8028, 0xc820ea2c00, 0x400, 0x400, 0x0, 0x0, 0x0)
/apps/lib/go/src/net/net.go:172 +0xe4
crypto/tls.(_block).readFromUntil(0xc820511410, 0x2b20e88c8b80, 0xc82a2a8028, 0x5, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(_Conn).readRecord(0xc820a99800, 0x9b4117, 0x0, 0x0)
/apps/lib/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820a99800, 0xc8224e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/apps/lib/go/src/crypto/tls/conn.go:939 +0x167

goroutine 13367415 [select]:
net/http.(_persistConn).writeLoop(0xc8202a0750)
/apps/lib/go/src/net/http/transport.go:1277 +0x472
created by net/http.(_Transport).dialConn
/apps/lib/go/src/net/http/transport.go:858 +0x10cb
What did you expect to see?
http post data successfully

What did you see instead?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@woria

This comment has been minimized.

Copy link
Author

commented Jun 16, 2016

@davecheney

    func (this *instance) pushMultiMsg(pushList) {
        defer func() {
            this.model.Close()
            if err := recover(); err != nil {
                logs.Logger.Critical(err)
            }
        }()

        pushChans := make(chan bool, 10000)

        for {
            //get userId from redis list
            userId := this.model.PopList(pushList)

            if userId == "" {
                // task finish
                break
            }
            pushChans <- true
            go this.concurrentPush(userId)
        }
        waitchan(pushChans)
    }

    func (this *instance) concurrentPush(userId) {
        client := &http.Client{}
        req, err := http.NewRequest("POST", 'http://localhost/send', bytes.NewReader(data))
        req.Close = true
        req.Header.Set("Content-Type", "application/json; charset=utf-8")
        resp, err := client.Do(req)
        defer resp.Body.Close()

        body, err := ioutil.ReadAll(resp.Body)
        if err != nil {
            return 
        }
        .....
    }

    func waitchan(pushChans chan bool) {
        for len(pushChans) > 0 {
            time.Sleep(time.Millisecond * 500)
        }
    }

Redis infrastructure:

LVS----twemproxy----REDIS

Redis list key structure:

push_list:abc

My program logic flow:

a) get userId from redis list [500000 msg data in list]

b) use a new goroutine for post data to URL [total 10000 goroutine]
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2016

@woria Please post a complete code sample the demonstrates the problem. This program must be executable so we can attempt to reproduce the problem you have reported.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2016

I'm sorry, I don't understand what the problem is. And I don't understand what the stack trace is showing us; how did you generate that trace? If the program crashed, it should have said more than that. What makes this a "net.runtime_pollWait error"? What is the error?

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2016

Don't create a new http.Client for each request. If you are sending 500,000 requests with a concurrency of 10,000 goroutines, you're very quickly going to run out of ephemeral ports on the client. http.Client objects are safe for use across multiple goroutines, so you should try sharing a single one. (Or, possibly one per goroutine; I don't know if you can issue 10,000 requests simultaneously from a single Client, or if there would be blocking.)

@woria

This comment has been minimized.

Copy link
Author

commented Jun 17, 2016

@quentinmit
I am sending 500,000 requests with a concurrency of 10,000 goroutines that will getting an EOF on HEAD/POST requests because of a Golang bug :#15723, I don't know how to fix this because this bug will fix on 1.8:#15723. so I am try to create a new http.Client for each post request.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2016

@woria
You can work around #15723 by putting a retry loop around the call to client.Do.

That said, you're also specifying request.Close, which is going to cause a new socket to be used for each request regardless of whether you reuse the client. Why are you specifying Close?

@woria

This comment has been minimized.

Copy link
Author

commented Jun 17, 2016

@quentinmit

Send HTTP POST requests to a host that seems to close the connection some specific time after the response. We see the error Post http://example.com: net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: EOF multiple times per second when doing around 100 requests per second.
It's hard to debug on live traffic and I haven't been able to reproduce this in a clean example.

issues: #15446

it should be getting an EOF on HEAD/POST requests where putting a retry loop around the call to client.Do.

This makes this trivial to fix, but it's too late for Go 1.7.
Easy to switch early in Go 1.8. Could also use a few more tests.

The Go version is 1.6 what I use in my program. It's the reason why I try to specifying request.Close that can keep the connection open

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Jun 23, 2016

issues: #15446

it should be getting an EOF on HEAD/POST requests where putting a retry loop around the call to client.Do.

It sounds like you're saying that you get a spurious error if the keepalive'd connection was closed before the request. That will be fixed in issue #15723 by retrying the request. In the meantime, however, your code (in concurrentPush) can retry the request when it receives that error. That will allow you to reuse the same connection for many requests.

The Go version is 1.6 what I use in my program. It's the reason why I try to specifying request.Close that can keep the connection open

We are never going to fix this for Go 1.6. If you are using Go 1.6, you are going to need to work around this as I suggest above. Otherwise, you can wait for a fix in Go 1.8.

There's fundamentally nothing we can do in Go. You are requesting a separate TCP connection for each request, and there are a finite number of TCP ports which can be used for that connection (65,535 theoretical ports, though the actual number is lower depending on how your kernel is configured). When those ports are all used up, it's not possible to make a new connection to the same host. You need to wait for those ports to be freed up before you can make new requests. You can look at this by running "netstat -n"; you'll see a separate connection in TIME_WAIT for each request. I suggest Googling "tcp time_wait linux" to find several great explanations of why this is.

I'm going to close this issue as I don't think there's anything we can do in Go. Please leave a comment if you think there is still a bug here that we can fix. Thanks.

@quentinmit quentinmit closed this Jun 23, 2016

@golang golang locked and limited conversation to collaborators Jun 23, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.