net/http: ListenAndServeTLS performs badly on Windows

[gopherbot]

by raheelgup:

Hi,

This is in reference to the crypto/tls package which is very slow and CPU intensive than
it should be.
I had been discussing this here :
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/QTDzrcDQmmw

I am using the following example as on the main go docs :
http://golang.org/pkg/net/http/#ListenAndServeTLS

I did a performance check on Windows 7 64 Bit for HTTPS connections and it is surprising
that it really performs badly.
My test methodology is simple, press F5 continously in chrome.
The CPU shot to 25-50% and the RAM also kept on increasing.
During the constant refresh, there are many "Connection was reset" messages. 

As against this the normal ListenAndServe example is very stable. I did an ab.exe -n
10000 -c 1000 http://127.0.0.1:PORT/index.html and it completes it in 60-70 seconds. The
MAX CPU touches 9-10% and RAM usage increases a little BIT and after the requests are
complete, it reaches near the previous RAM usage.

Adam from Golang also did a profiling :
(pprof) top10 -cum
Total: 1075 samples
       0   0.0%   0.0%      827  76.9% schedunlock
       0   0.0%   0.0%      822  76.5% net/http.(*conn).serve
       0   0.0%   0.0%      813  75.6% crypto/tls.(*Conn).Handshake
       0   0.0%   0.0%      813  75.6% crypto/tls.(*Conn).serverHandshake
       2   0.2%   0.2%      578  53.8% crypto/elliptic.(*CurveParams).ScalarMult
       0   0.0%   0.2%      495  46.0% crypto/tls.(*ecdheRSAKeyAgreement).generateServerKeyExchange
       9   0.8%   1.0%      449  41.8% math/big.nat.div
     135  12.6%  13.6%      431  40.1% math/big.nat.divLarge
       8   0.7%  14.3%      347  32.3% math/big.(*Int).Mod
       2   0.2%  14.5%      343  31.9% math/big.(*Int).QuoRem

So 75% of the time is being taken in handshaking and the rest is probably taken in
dealing with the garbage from that.

SO I am wondering if this can be fixed in the next GO Version.

Regards,
R

[alexbrainman]

Comment 1:

Thank you for taking time to create the issue. But I see no new information for me to go
on - you are just repeating what you have already said on go-nuts. Perhaps, others might
have some ideas.
Alex

[gopherbot]

Comment 2 by raheelgup:

Hi,
AGL asked me to open the issue here and hence I did.
If you want me to run any test, I would be happy to.
AGL did say he is convinced of the CPU Usage going high as its taking 75% of the
resources to do a TLS Handshake.
Regards,
R

[alexbrainman]

Comment 3:

R,
I think the only way for you to advance your issue is to convince us that there is a
problem here: bug in code somewhere, leaking resources, great inefficiencies. It is on
you to demonstrate that. Best way to do it, I think, is to write a Go test that
establishes connection between SSL client and server - all in one test. You could run
this test in a loop to measure performance. You could use runtime memory stats to show
memory increases. You could use benchmarking tools in test package to measure your
findings. Please, have a go and report your findings.
Alex

[davecheney]

Comment 4:

Status changed to WaitingForReply.

[rsc]

Comment 5:

Labels changed: added go1.1.

[gopherbot]

Comment 6 by erlang@rambocoder.com:

On Ubuntu, the ListenAndServerHTTPS performs badly too.
siege --benchmark --concurrent=100 "https://localhost:8082"
gives 21.01 trans/sec while to a similar hello world node.js app, which uses the same
SSL certs, the benchmark gives 508.64 trans/sec.

[alexbrainman]

Comment 7:

Please, provide details of your setup, so I can try and reproduce these here. Go program
you test, how you create certificates - everything you think is needed to reproduce your
details. Thank you.
Alex

[gopherbot]

Comment 8 by erlang@rambocoder.com:

I generated the keys with:
openssl genrsa 2048 > key.pem
openssl req -new -x509 -key key.pem -out cert.pem -days 1095
and the code that was benchmark on dual core i7 with hyperthreading is:
package main
import (
    "log"
    "net/http"
    "runtime"
)
func handler(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    w.Write([]byte("This is an example server.\n"))
}
func main() {
    runtime.GOMAXPROCS(runtime.NumCPU())
    http.HandleFunc("/", handler)
    log.Printf("About to listen on 8082. Go to https://127.0.0.1:8082/")
    err := http.ListenAndServeTLS(":8082", "cert.pem", "key.pem", nil)
    if err != nil {
        log.Fatal(err)
    }
}

[alexbrainman]

Comment 9:

Thank you for your instructions.
I tested my both (linux and windows) computers. And they give similar figures:
linux:
Lifting the server siege...      done.
Transactions:                   1867 hits
Availability:                 100.00 %
Elapsed time:                 118.75 secs
Data transferred:               0.05 MB
Response time:                  6.23 secs
Transaction rate:              15.72 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                   97.89
Successful transactions:        1867
Failed transactions:               0
Longest transaction:           13.59
Shortest transaction:           0.32
windows:
Lifting the server siege...      done.
Transactions:                    779 hits
Availability:                 100.00 %
Elapsed time:                  40.56 secs
Data transferred:               0.02 MB
Response time:                  4.85 secs
Transaction rate:              19.21 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                   93.11
Successful transactions:         779
Failed transactions:               0
Longest transaction:           10.22
Shortest transaction:           0.34
Alex

[gopherbot]

Comment 10 by erlang@rambocoder.com:

Alex, I was just chiming in to say that the performance of ListenAndServerTLS appears to
be as slow on linux as it is on on Windows. My ideal web server setup with Go is having
all SSL without any type of SSL accelerator in front of my Go web app. The current
implementation calls for having a very beefy server if there is a great load on my Go
web app.

[alexbrainman]

Comment 11:

erlang,
I am not arguing that performance of ListenAndServerTLS is good or bad. All I am saying
that there is not much difference between using linux and windows.
Alex

[gopherbot]

Comment 12 by erlang@rambocoder.com:

Alex, I agree with you.
-rambocoder

[alexbrainman]

Comment 13:

created new issue https://golang.org/issue/4299 for data I see
myself.

Status changed to Duplicate.

Merged into issue #4299.

[gopherbot]

Comment 14 by morapeter:

Let me write down my benchmarks and a temporary fix.
On my laptop (with core 2 duo) I measure the Go server's performance by
time ./myServer
and watched the row "user" in the result. The server only establish the connection. I
tested it with the following benchmark:
for ((i=0;i<100;i++)); do curl --insecure https://127.0.0.1/test& done
The results:
without cipher: 8 sec CPU load (it is using ECDHE)
with --ciphers AES256-SHA (or any other without ECDHE): 2 sec
with --ciphers ECDHE-RSA-AES128-SHA (or any other with ECDHE): 8 sec
So I modified the src/pkg/crypto/tls/handshake_server.go to prefer non-ECDHE handshake
if it's possible. In readClientHello(...) function I replaced
for _, id := range preferenceList {
    if hs.suite = c.tryCipherSuite(id, supportedList, c.vers, hs.ellipticOk, hs.ecdsaOk); hs.suite != nil {
        break
    }
}
with
// We prefer everything which is not elliptic
for _, id := range preferenceList {
    if hs.suite = c.tryCipherSuite(id, supportedList, c.vers, false, hs.ecdsaOk); hs.suite != nil {
        break
    }
}
// If there is no match, let's allow elliptic
if hs.suite == nil && hs.ellipticOk {
    for _, id := range preferenceList {
        if hs.suite = c.tryCipherSuite(id, supportedList, c.vers, hs.ellipticOk, hs.ecdsaOk); hs.suite != nil {
            break
        }
    }
}
With this fix I have:
without cipher: 2 sec CPU load (because we prefer non-ECDHE handshake now)
with --ciphers AES256-SHA (or any other without ECDHE): 2 sec
with --ciphers ECDHE-RSA-AES128-SHA (or any other with ECDHE, the server still support
it if no other handshake is possible): 8 sec
By the way, I believe that the handshake should be around 0.5 sec (same machine,
openssl).