Persistent connections to volume servers through FUSE

[Woellchen]

I've got one question regarding FUSE mounts and connections to volume servers.
As far as I could see on my server/s and from the code, retrieving or saving files to volume servers happens in dedicated HTTP requests. This is probably the case for all volume server operations.
Would it not be possible to keep the connections alive all the time? Or have some kind of connection pooler that keeps a bunch of idle connections that can be reused?
Making a new request each and every time is very expensive and adds a lot of overhead. It degrades the overall performance. Especially when the connections are also encrypted (TLS handshake..).
Did I miss something or could this be a nice feature for SeaweedFS?

[kmlebedev]

@chrislusf It seems enough to enable the reuse connection by means of tcp
See
https://github.com/zalando/skipper/pull/1511/files

How does curl

curl -sv -o /dev/null http://www.google.com http://www.google.com  2>&1| grep -i 'connection'
* Connection #0 to host www.google.com left intact
* Re-using existing connection! (#0) with host www.google.com
* Connection #0 to host www.google.com left intact

[chrislusf]

Current code does try to exhaust the responses https://github.com/chrislusf/seaweedfs/blob/master/weed/util/http_util.go#L373

Maybe the per host connection count is too high? https://github.com/chrislusf/seaweedfs/blob/master/weed/util/http_util.go#L24

[kmlebedev]

Current code does try to exhaust the responses https://github.com/chrislusf/seaweedfs/blob/master/weed/util/http_util.go#L373

Maybe the per host connection count is too high? https://github.com/chrislusf/seaweedfs/blob/master/weed/util/http_util.go#L24

Need to test
OJ/gobuster#127
MaxIdleConnsPerHost == number of goroutines/'threads'
and
http://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/

[kmlebedev]

try set MaxIdleConns = MaxIdleConnsPerHost

	Transport = &http.Transport{
		MaxIdleConns: 1024,
		MaxIdleConnsPerHost: 1024,
	}

For example you have 10 volume server, then there are 10 connections per server

[chrislusf]

adjusted with 487e435

but how to verify the fix?

[Woellchen]

Shall I test something? I can build a new binary and run it.
Meanwhile I did some tcpdumping. It turns out that it is always the volume server that closes the connection. It sends FINACK and mount returns with FINACK as well, followed by a last ACK from the volume server. So a normal connection close.
Maybe there is something on the volume server side that we can do?

[kmlebedev]

Shall I test something? I can build a new binary and run it.
Meanwhile I did some tcpdumping. It turns out that it is always the volume server that closes the connection. It sends FINACK and mount returns with FINACK as well, followed by a last ACK from the volume server. So a normal connection close.
Maybe there is something on the volume server side that we can do?

You incorrectly mean FastOpen for Socket but I have not found how to enable it for net/http
But I found https://github.com/valyala/fasthttp

[Woellchen]

Hm, what I really mean is how can we keep connections up without dying? At least for some configurable amount of time.
I just see that connections are closed after a few seconds (2 rounds of 15sec TCP keepalive?) and I want to keep them open for at least a few minutes.
Can we maybe use this: https://golang.org/pkg/net/#TCPConn.SetKeepAlive and https://golang.org/pkg/net/#TCPConn.SetKeepAlivePeriod?

[chrislusf]

@Woellchen try to adjust weed volume -idleTimeout=30

Existing code also disconnects if there are no data traffic. https://github.com/chrislusf/seaweedfs/blob/master/weed/util/net_timeout.go#L58

[Woellchen]

@chrislusf thanks! That actually helped. I still see degraded performance when loading files from time to time, so the root cause were not the kept-alive TCP connection.
I will try to do some debugging and find out what is wrong in my case.

[chrislusf]

Do you configure concurrent writers? Make sure it does not create too many goroutines.

weed mount -concurrentWriters=128

  -concurrentWriters int
    	limit concurrent goroutine writers if not 0 (default 128)