Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: Conn.rawInput buffer has no chance to shrink on large number of idle conns #43563

Open
cch123 opened this issue Jan 7, 2021 · 10 comments · May be fixed by #48229
Open

crypto/tls: Conn.rawInput buffer has no chance to shrink on large number of idle conns #43563

cch123 opened this issue Jan 7, 2021 · 10 comments · May be fixed by #48229
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@cch123
Copy link
Contributor

cch123 commented Jan 7, 2021

What version of Go are you using (go version)?

go version go1.14.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

Open TLS on HTTP server.

TLS HTTP server:

package main

import (
	"bufio"
	"crypto/tls"
	"fmt"
	"io/ioutil"
	"net"
	"net/http"
	"time"

	_ "net/http/pprof"
)

func init() {
	go http.ListenAndServe(":9999", nil)
}

func main() {
	l, err := net.Listen("tcp4", ":1234")
	if err != nil {
		fmt.Println(err)
		return
	}

	cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		fmt.Println(err)
		return
	}

	for {
		c, err := l.Accept()
		if err != nil {
			return
		}

		go func() {
			c = tls.Server(c, &tls.Config{
				Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true,
			})

			if err != nil {
				fmt.Println(err)
				return
			}

			r := bufio.NewReader(c)
			for {
				c.SetReadDeadline(time.Now().Add(time.Second * 5))
				req, err := http.ReadRequest(r)
				if err != nil {
					if e, ok := err.(net.Error); ok && e.Timeout() {
						continue
					}
					c.Close()
					return
				}

				_, err = ioutil.ReadAll(req.Body)
				if err != nil {
					fmt.Println(err)
					return
				}

				// write respose
				resp := &http.Response{ProtoMajor: 1, ProtoMinor: 1, StatusCode: http.StatusOK, Header: http.Header{}, Body: http.NoBody}
				err = resp.Write(c)
				if err != nil {
					fmt.Println(err)
					c.Close()
					return
				}
			}
		}()
	}
}

TLS HTTP client.

package main

import (
	"crypto/tls"
	"fmt"
	"bytes"
	"io/ioutil"
	"net/http"
	"os"
	"strconv"
	"sync"

	_ "net/http/pprof"

	"go.uber.org/ratelimit"
)

func init() {
	go http.ListenAndServe(":19999", nil)
}

func main() {
	url := os.Args[3]
	connNum, err := strconv.ParseInt(os.Args[1], 10, 64)
	if err != nil {
		fmt.Println(err)
		return
	}

	qps, err := strconv.ParseInt(os.Args[2], 10, 64)
	if err != nil {
		fmt.Println(err)
		return
	}

	bucket := ratelimit.New(int(qps))

	var l sync.Mutex
	connList := make([]*http.Client, connNum)

	for i := 0; ; i++ {
		bucket.Take()
		i := i
		go func() {
			l.Lock()
			if connList[i%len(connList)] == nil {
				connList[i%len(connList)] = &http.Client{
					Transport: &http.Transport{
						TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
						IdleConnTimeout:     0,
						MaxIdleConns:        1,
						MaxIdleConnsPerHost: 1,
					},
				}
			}
			conn := connList[i%len(connList)]
			l.Unlock()
			if resp, e := conn.Post(url, "application/json", bytes.NewReader(make([]byte, 16000))); e != nil {
				fmt.Println(e)
			} else {
				defer resp.Body.Close()
				ioutil.ReadAll(resp.Body)
			}
		}()
	}

}

./client 40000 1000 https://ip:1234/

image

This TLS HTTP server costs about 2.2GB RSS.

To reduce the memory cost, we shrink the TLS read buffer when read timeout:

TLS Read function:		
....
if e, ok := err.(net.Error); ok && e.Timeout() {
    if Conn.rawInput.Len() == 0 && Conn.input.Len() == 0 && Conn.hand.Len() == 0 {
        c.rawInput = *bytes.NewBuffer(make([]byte, 0, bytes.MinRead))
    }
}
....

And the memory usage decreased from 2.2GB to around 560 MB

But I don't know whether this is a proper fix.

If it is. I'm happy to open a PR for this

@cch123 cch123 changed the title crypto/tls : rawInput buffer has no chance to shrink on large number of idle conns crypto/tls : Conn.rawInput buffer has no chance to shrink on large number of idle conns Jan 7, 2021
@toothrot toothrot changed the title crypto/tls : Conn.rawInput buffer has no chance to shrink on large number of idle conns crypto/tls: Conn.rawInput buffer has no chance to shrink on large number of idle conns Jan 8, 2021
@toothrot
Copy link
Contributor

toothrot commented Jan 8, 2021

/cc @FiloSottile

@toothrot toothrot added this to the Backlog milestone Jan 8, 2021
@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 8, 2021
@FiloSottile
Copy link
Contributor

FiloSottile commented Jan 8, 2021

I assume this is after the #42035 fix?

@cch123
Copy link
Contributor Author

cch123 commented Jan 9, 2021

I assume this is after the #42035 fix?

Yes

@jvilhuber
Copy link

jvilhuber commented Dec 1, 2021

I'm very interested in this topic as well. Any progress on this issue and the linked PR?

@cch123
Copy link
Contributor Author

cch123 commented Dec 3, 2021

I'm very interested in this topic as well. Any progress on this issue and the linked PR?

I don't have a good enough server to test this PR, maybe the bench data is not convincing

@jkralik
Copy link

jkralik commented Dec 7, 2021

@FiloSottile Is there any news? Because I have the same issue with golang 1.17.3 .

jkralik added a commit to plgd-dev/hub that referenced this issue Dec 8, 2021
jkralik added a commit to plgd-dev/hub that referenced this issue Dec 8, 2021
jkralik added a commit to plgd-dev/hub that referenced this issue Dec 8, 2021
jkralik added a commit to plgd-dev/hub that referenced this issue Dec 8, 2021
* coap-gateway: update go-coap

* docker: patch tls.Conn for shrink connection buffer

golang/go#43563

* fix for CR
@gopherbot
Copy link

gopherbot commented Dec 9, 2021

Change https://golang.org/cl/370581 mentions this issue: crypto/tls: shrink tls Conn's rawInput with a small buffer

@jvilhuber
Copy link

jvilhuber commented May 9, 2022

@FiloSottile Any update?

@askuy

This comment was marked as duplicate.

@JellyZero
Copy link

JellyZero commented Sep 19, 2022

@FiloSottile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants