Queueing service stack doesn't end the queue

[scottcorgan]

If a service ends the request, the queue it was a part of remains in memory intermittently interferes with requests.

Looking into converting the service stack building middleware to use the stacked module.

[scottcorgan]

Benchmarks of pre composed stack vs dynamically built stack per request (20 seconds, 1 thread, 400 connections):

No middleware

Test 1

Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    43.64ms    2.41ms  64.58ms   74.77%
    Req/Sec     9.23k   417.08    10.09k    74.26%
  183029 requests in 20.00s, 28.45MB read
  Socket errors: connect 0, read 19, write 1, timeout 0
  Non-2xx or 3xx responses: 183029
Requests/sec:   9151.49
Transfer/sec:      1.42MB

Test 2

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    44.10ms    2.34ms  59.05ms   72.90%
    Req/Sec     9.13k   405.97    10.01k    74.44%
  181035 requests in 20.00s, 28.14MB read
  Non-2xx or 3xx responses: 181035
Requests/sec:   9051.78
Transfer/sec:      1.41MB

Full built stack (Current state)

Test 1

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   290.49ms   23.09ms 511.00ms   76.25%
    Req/Sec     1.39k   143.10     1.83k    80.65%
  27537 requests in 20.02s, 17.88MB read
  Socket errors: connect 0, read 7, write 0, timeout 0
Requests/sec:   1375.75
Transfer/sec:      0.89MB

Test 2

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   272.75ms    7.62ms 304.60ms   72.68%
    Req/Sec     1.47k    52.83     1.64k    82.86%
  29199 requests in 20.05s, 18.96MB read
Requests/sec:   1456.37
Transfer/sec:      0.95MB

Dynamically built stack with all modules

Test 1

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   300.63ms   10.84ms 342.33ms   78.39%
    Req/Sec     1.33k    81.88     1.57k    83.87%
  26400 requests in 20.05s, 17.15MB read
Requests/sec:   1316.88
Transfer/sec:      0.86MB

Test 2

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   291.56ms    6.33ms 313.65ms   70.76%
    Req/Sec     1.38k    34.44     1.47k    87.88%
  27200 requests in 20.01s, 17.67MB read
Requests/sec:   1359.35
Transfer/sec:      0.88MB

Dynamically built stack with limited modules

Test 1

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   285.00ms   14.44ms 327.25ms   63.79%
    Req/Sec     1.40k   101.91     1.46k    80.00%
  27999 requests in 20.04s, 18.18MB read
Requests/sec:   1397.08
Transfer/sec:      0.91MB

Test 2

 >  wrk -t1 -c400 -d20s http://localhost:3474
Running 20s test @ http://localhost:3474
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   291.83ms   12.52ms 336.88ms   72.76%
    Req/Sec     1.37k    95.45     1.50k    85.29%
  27201 requests in 20.01s, 17.67MB read
Requests/sec:   1359.13
Transfer/sec:      0.88MB