/count endpoint barfs under load

[wirtsi]

I am doing some load testing with the starter example. So for example

ab  -n 100000 http://localhost:9000/count                                                                                               
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:
Server Hostname:        localhost
Server Port:            9000

Document Path:          /count
Document Length:        5 bytes

Concurrency Level:      1
Time taken for tests:   206.992 seconds
Complete requests:      100000
Failed requests:        49210
   (Connect: 0, Receive: 0, Length: 49210, Exceptions: 0)
Total transferred:      40149210 bytes
HTML transferred:       549210 bytes
Requests per second:    483.11 [#/sec] (mean)
Time per request:       2.070 [ms] (mean)
Time per request:       2.070 [ms] (mean, across all concurrent requests)
Transfer rate:          189.42 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1 148.9      0   19570
Processing:     0    1   8.4      0    1024
Waiting:        0    1   8.0      0    1024
Total:          0    2 149.1      1   19571

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%  19571 (longest request)

It appears I can throw the /count endpoint off by setting a higher concurrency

ab -c12 -n 100000 http://localhost:9000/count                                                                                            
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16387 requests completed

This always happens at around 16k requests, even at a concurrency of only 2

Os is MacOs Sierra, Scala 2.12.2 (installed via brew), 2.6.x branch of this repo

Nothing too dramatic but still a bit of a surprise how quickly threading can barf, even with Scala :)

[slohr]

out of curiosity I tested this too with similar results

ab -c12 -n 100000 http://localhost:9000/count
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16405 requests completed

Then decided to run it under two shells while hitting /count via my browser.

first...

...
Benchmarking localhost (be patient)
Completed 10000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 13774 requests completed

second...

...
Benchmarking localhost (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 2626 requests completed

The browser requests returned the updated counts caused by the ab commands and continued to work after the ab commands barfed. Surprising the ab commands failed but heartening that concurrent access with Chrome worked fine.

[wirtsi]

True, good thing the browser still gets data. I just found it strange that the concurrency causes some sort of deadlock (I would assume ... even with your two ab scripts running you get until around 16k), I thought this whole actor thing was to avoid exactly this....

[marcospereira]

ab is known for having some problems.

Could you please test with wrk?

[wsargent]

This always happens at around 16k requests, even at a concurrency of only 2

Please see the performance test framework results at https://playframework.github.io/prune/ -- this is far more likely to be a consequence of doing a load test on MacOs Sierra, with the load testing framework on the machine itself, using apache bench.

Try on a standalone Linux machine and use gatling or wrk: see https://developer.lightbend.com/guides/play-rest-api/appendix.html#load-testing for details.