define couchDB connection pool size #654
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of change
Description
When the CouchDB is used as the StateDB, we observe too many TCP connections at the
time_wait
state.As the
/proc/sys/net/ipv4/tcp_fin_timeout
(i.e., keep-alive timeout) is usually set to 60 seconds in the Linux server, we quickly get into the following error when the peer tries to access the CouchDB:connect: cannot assign requested address
Reason for the above behavior. The peer creates a large number of TCP connections to CouchDB because the default reusable connection pool size is 2 per host while the max pool size is 100 (combining all hosts) -- refer to transport.go and golang http doc. Thus only 2 TCP connections can be reused by the peer when it accesses CouchDB. When more than two goroutines in the peer try to simultaneously access the CouchDB, it results in new non-reusable connections (i.e., creation of new TCP connection outside the pool). Hence, when the peer runs for around 30 seconds at a high load, it results in too many TCP connections at the
time_wait
state.Fix: We set
MaxIdleConns
andMaxIdleConnsPerHost
to 2000. This would create 2000 TCP connections in the pool and it would be reused whenever the peer accesses the CouchDB. As the peer is the only host for the CouchDB, we set the same value for both parameters.Additional details
We tested it using a load generator by generating 700 tps. Now, there is no TCP connection at the
time_wait
state. This PR is cherry-picked from the master -- #429Related issues
FAB-17277