possible HTTP load problem

[akka-ci]

Issue by rkuhn
Wednesday May 06, 2015 at 09:33 GMT
Originally opened as akka/akka#17395

---

reported on twitter by Niko Will (@win1imb), reproducer is at https://github.com/win1imb/akka-poc

[akka-ci]

Comment by n1ko-w1ll
Wednesday May 06, 2015 at 11:59 GMT

---

Hi and thanks for opening this issue. To reproduce the problem it should be enough to run the SearchService class and the corresponding Gatling test. On my machine (Windows 7, 64 Bit, 16GB RAM) around 40% of the requests end up in an error message saying

java.net.ConnectException: Connection refused: no further information: localhost/127.0.0.1:8080

[akka-ci]

Comment by rkuhn
Wednesday May 06, 2015 at 12:15 GMT

---

Would you mind using the akka.http.javadsl.Http extension directly and specifying a larger value for the backlog parameter? This one defaults to 100 and if connections cannot be accepted quickly enough this will give you the observed result.

[akka-ci]

Comment by jrudolph
Wednesday May 06, 2015 at 13:33 GMT

---

@rkuhn that's also a good reminder to add the missing Java overloads for specifying settings.

@win1imb The result of such a load test will very much depend on benchmarking tools and setup.

"10000 concurrent users" can mean a lot of different things. E.g. if you setup wrk to use 10000 connections, it will try to create all those (persistent HTTP) connections but at the same time it already starts hammering the server with requests. Normally wrk will be able to completely max out server CPU by sending just on 8 connections (or at least as many as CPUs are available) if handling the request is completely CPU-bound which is the usual benchmarking case.

Increasing the number of connections from this point will only mean that connection establishment and request handling will battle for resources which means that timeouts are likely.

So, in summary, try to separate testing for performance (How many RPS are possible?) and scalability over connections (What is the performance impact of running requests over more connections?). From what I've seen scalability is ok while performance still needs to be improved.

/cc @sirthias

[akka-ci]

Comment by n1ko-w1ll
Thursday May 07, 2015 at 06:34 GMT

---

@rkuhn I had a quick look but I'm familiar with the internals of akka. If I've understood it correctly, I have to use the Http.get(system).bind(....) method, right? The one where I can specify the backlog size as parameter has a lot of other parameters I have no clue about, what to pass here.

@jrudolph you're right, that's why I provided the stress test setup with gatling in my example project, too. As you said, increasing the number of connections will end up in a battle for resources. But if you have to choose a technology, one can cope with that and answer the requests even if not very fast and the other refuses the connections because it's not able to manage the battle for resources, which one would you use? In a world with SLAs and zero downtime, answering a request with 150ms (which was the mean of spring boot in my comparison) compared to 40% connection refused, I would choose the first option ;)

Don't get me wrong, I like akka and actor based programming, especially what it does to handle concurrency and resilience in a very concise way. As I said, possibly the error sits in front of the computer. But at the end of the day, we have to be confident with the technology we use.

[akka-ci]

Comment by n1ko-w1ll
Thursday May 07, 2015 at 06:38 GMT

---

BTW, had to change my username, so the code examples are now at https://github.com/n1ko-w1ll/akka-poc

[akka-ci]

Comment by rkuhn
Thursday May 07, 2015 at 11:38 GMT

---

My plan was to look into this today, but might not be able to, depending on how the traveling goes. I’ll definitely try out your project to see what’s wrong.

[akka-ci]

Comment by jrudolph
Friday May 08, 2015 at 08:03 GMT

---

@n1ko-w1ll

But if you have to choose a technology, one can cope with that and answer the requests even if not very fast and the other refuses the connections because it's not able to manage the battle for resources, which one would you use?

This assumes that the problem is related to latency (i.e. the processing pipeline is too long but the server has still some CPU capacity left). If you assume the problem is throughput related (i.e. the server just cannot keep up with work) then the result is not slow responses but a DOS scenario where response times will increase and eventually the server won't answer any more at all. If the problem is throughput-related then in spirit of reactivity you would actually choose the server that backpressures new connections (e.g. by letting connection attempts timeout and rely on them reconnecting with a backoff strategy) instead of a server which lets itself get overwhelmed with connections/requests.

What I meant before with "From what I've seen scalability is ok while performance still needs to be improved." is that akka-http may currently be just too slow to run your example but it doesn't really depend on the number of connections. So, I suspect that all that you are seeing is a consequence of akka-http being too slow.

That said, I agree with you that akka-http should be at least in the same ballpark performance-wise as spring. :)

Btw. thanks for including the gatling script and sorry that I overlooked it before.

[ktoso]

Linked reproducer is not available and we did a lot of work recently. Reopen if needed