Concurrency throttling provides stable response time when faced with un-predictable concurrency

[robert-s-lee]

Exalate commented:

Overview

Microservices and web applications user concurrency is often not predictable. Throttling the concurrency into the database at tested concurrency can stabilize the throughput and response in the face of unpredictability.

Details

Concurrency tests that exceed server's resource capacity worsen's performance. By throttling concurrency at about 75% of the resource's limit, better performance can be achieved. Throttling the concurrency limits the context switching that has a negative effect on the performance.

Results

The graphs below show throughput and response time by

• varying concurrency from 1 to 8
• throttling concurrency into the database at 3 and no throttling
• the blue line represents concurrency throttled at 3
• the red line represents no throttling

By capping the concurrency into the database, the throughput is capped to that of concurrency=3

By capping the concurrency into the database, the response time shows stable 1ms even at high concurrency whereas, without the throttling, the response time gets worse and worse.

Test Config

Below script was run varying the concurrency from 1 to 8 on i3 CPU with Linux. The workload was set to 100% read which will stress the CPU. Concurrency of 3 is expected to be the max for this system.

for c in 1 2 3 4 5 6 7 8; do
echo $c
kv -read-percent 100 -duration 20s -concurrency $c 'postgresql://root@localhost:26256?sslmode=disable'
sleep 5
done

haproxy.cfg was setup in two different ways.

default which does not limit concurrency

global
  maxconn 4096

defaults
    mode                tcp
    timeout connect     10s
    timeout client      1m
    timeout server      1m

listen psql
    bind :26256
    mode tcp
    balance roundrobin
    server cockroach1 127.0.0.1:26257 

limit concurrency to 3 into the database

global
  maxconn 4096

defaults
    mode                tcp
    timeout connect     10s
    timeout client      1m
    timeout server      1m

listen psql
    bind :26256
    mode tcp
    balance roundrobin
    server cockroach1 127.0.0.1:26257 maxconn 3

Jira Issue: DOC-146

[bdarnell]

With 8 clients and maxconn 3, what happened to the clients that were not allowed connections? Did they get to take a turn or did they just get blocked waiting for a connection?

PGBouncer may be better than HAProxy here, since it (in "transaction pooling" mode) demultiplexes the incoming requests across different backend connections.

[robert-s-lee]

HAproxy doc indicates the client waits for the next available slot. The kv load generator did not report any error during the test. Just looking at PGBouncer -- worth testing and measuring the characteristics.

[petermattis]

@robert-s-lee How were you measuring response times in your setup? It makes absolute sense that limiting the concurrent connections to 3 will stabilize the server side response times, but the client side latencies should see a similar rise in both situations. That is exactly what I see in the various "scalability" graphs I produce. When we start hitting the throughput limiting, increasing concurrency results in diminishing returns for throughput but latencies continue to rise.

[robert-s-lee]

https://docs.google.com/spreadsheets/d/1BZ9PTrsgTzSl2vUyjq8CzT6EzcHsAZRSasjpvYZt9R8/edit#gid=2103637363 has the raw data of KV output. The response time is p99 (on the graph). p95 is also in the spreadsheet. In this test, KV and CRDB are on the same system. I have been meaning to rerun so that 1) KV and CRDB are on separate systems to see if the curve look different 2) run longer with bigger data set 3) also do write as the previous is 100% read in order to mimic a specific scenario 4) try out PGBouncer during the re-run.

[petermattis]

@robert-s-lee What linux box were you running on? "i3 CPU" is vague.

[petermattis]

No need to answer the question about the linux box, I understand what is going on. The haproxy maxconn parameter isn't doing what we want it to do here and is causing kv to report inaccurate numbers. Consider this haproxy config:

global
  maxconn 4096

defaults
    mode                tcp
    # Timeout values should be configured for your specific use.
    # See: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-timeout%20connect
    timeout connect     10s
    timeout client      1m
    timeout server      1m
    # TCP keep-alive on client side. Server already enables them.
    option              clitcpka

listen psql
    bind :26000
    mode tcp
    balance roundrobin
    server cockroach1 localhost:26257 check maxconn 1

If I run kv --read-percent=100 --concurrency=1 'postgres://root@localhost:26000?sslmode=disable' I see a steady stream of output:

~ kv --read-percent=100 --concurrency=1 'postgres://root@localhost:26000/?sslmode=disable'
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
      1s        0         5087.9         5087.8      0.2      0.3      0.3      1.1
      2s        0         5115.3         5101.6      0.2      0.3      0.3      1.2
      3s        0         5116.5         5106.5      0.2      0.3      0.3      1.1
      4s        0         5174.2         5123.4      0.2      0.3      0.3      0.6
      5s        0         5138.5         5126.5      0.2      0.3      0.3      1.1

But if I run a second kv using the same parameters while the first is still running I see:

~ kv --read-percent=100 --concurrency=1 'postgres://root@localhost:26000/?sslmode=disable'
<10s pause>
2018/02/25 20:11:13 EOF

What happened? The second kv opened a new connection which exceeded maxconn. So haproxy queued the connection. But since the first kv never released its connection, we starved the second kv. The 10s pause before the EOF error corresponds to the timeout connect in our haproxy config. I verified this by bumping that timeout to 20s and the pause changed from 10s to 20s as expected.

@robert-s-lee I think if you try and reproduce this you'll see EOF messages like I saw above. The reason the response times were not increasing with concurrency when you had maxconns capped at 3 was because the other clients were effectively blocked and not making any progress at all.

Instead of using haproxy to limit concurrency, we can do so in kv itself by adjusting db.SetMaxOpenConns(). I haven't tested that yet, but I'm pretty confident that it will show latencies (response times) increasing as we increase concurrency.

[petermattis]

As a follow-up, I extended kv with --connections flag to limit the number of connections to the database (the previous kv behavior was to allow a connection per worker). In the graph below, the red line is without limiting the number of connections and the blue line is with --connections=8:

I ran this experiment on 2 n1-standard-4 GCE machines (4 vCPU machines). Throughput flatlines at concurrency=8 and at that point latencies begin to rise. But notice that allowing more concurrent connections (the red line) provides better (lower) latency than restricting the number of concurrent connections.

This is a single experiment. We've seen with TPC-C that increased concurrency can slow the system down. It is possible that with higher concurrency values, limiting the number of connections would be beneficial. It is possible that allowing lots of concurrent connections but limiting the number of concurrent queries would be beneficial.

[petermattis]

Ran another experiment where I adjusted the concurrency from 2 to 64 in increments of 2. The red line is with unlimited number of connections and the blue line with --connections=32:

I think what we're seeing here and in the previous experiment is lack of fairness in the database/sql connection pool. This is just a hypothesis at this point, but I'd be willing to bet that a request for a new connection doesn't end up on a queue, but waits on a condition variable and there is to assurance of fifo behavior.

[jseldess]

@petermattis, @robert-s-lee, I don't understand if there is known docs work here for 2.1. Can one of you clarify?

[petermattis]

@jseldess I don't think there is anything to document yet. I think it is likely that we should add some sort of admission control mechanism to query processing on the server side. So far my experimentation has shown that trying to do this on the client side is extremely problematic (see the blue latencies in the chart above).