Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apache (4 and 5) server engine high memory consumption under load #481

Open
strohel opened this issue Aug 26, 2020 · 3 comments
Open

Apache (4 and 5) server engine high memory consumption under load #481

strohel opened this issue Aug 26, 2020 · 3 comments

Comments

@strohel
Copy link

strohel commented Aug 26, 2020

Hi, when doing some benchmarking of various server engines (that also resulted in fixing #141, thx @daviddenton), I've found that apache4 and apache (5) engines' memory consumption rises sharply starting with around 64 concurrent connections. No other engines exhibit this issue.

See the graphs: https://storage.googleapis.com/strohel-pub/bench-http4k-server-engines-apachemem/bench-results.html (Max Memory Usage vs. Connections)

http4k runs in a Docker container with memory limit of 512 MiB, in OpenJDK 11 with -XX:MaxRAMPercentage=75 (allow Java heap up to 75% of assigned memory limit). The container is actually killed on Out-Of-Memory when (extreme) 1024 concurrent connections are fired at it.

http4k version: 3.258.0
code: https://gitlab.com/gooutopensource/locations-kt-http4k/-/tree/kt-http4k-apache4

I suspected this could be caused by the default socket backlog size of 1000, so I've tried with a lowered socket backlog of 100, with no effect on performance or memory consumption (the apache4-q100-* lines in the graphs).

This may not be a bug but rather a property, I'm filling this mainly to consult, also because apache(4) is otherwise the most performant engine.

@strohel strohel changed the title Apache server engine Apache (4 and 5) server engine high memory consumption under load Aug 26, 2020
@daviddenton
Copy link
Member

daviddenton commented Sep 1, 2020

We have now introduced Apache 4, 5 and Netty to the TechEmpower benchmarks. Latest results are here: https://www.techempower.com/benchmarks/#section=test&runid=203ff846-d210-4205-851f-611c5c246c8d

Unfortunately for some reason, Apache 5 is not starting in the test runs - I think this is to do with matching of the canonical hostname in the creation of the server, but can't yet work out how fix it (as it passes the TE build stage).

Netty looks good though... :)

@strohel
Copy link
Author

strohel commented Sep 2, 2020

Unfortunately for some reason, Apache 5 is not starting in the test runs - I think this is to do with matching of the canonical hostname in the creation of the server, but can't yet work out how fix it (as it passes the TE build stage).

Maybe I had a similar problem with Apache 5.0 backend locally: it would reply correctly on http://localhost:8080/, but would respond with HTTP 400 + "Not authoritative" in the body on http://127.0.0.1:8080/. I think the culprit is in:

https://github.com/apache/httpcomponents-core/blob/edeb19b1a7490f6753ce118ac7d471a974736be6/httpcore5/src/main/java/org/apache/hc/core5/http/protocol/RequestHandlerRegistry.java#L93-L109

The HttpComponents seem to try to do host-based virtualhosting here... (it seems that every thing called apache needs to do virtualhosting... ;) )

@daviddenton
Copy link
Member

daviddenton commented Sep 3, 2020

I think I've fixed this in the benchmarks by messing with the virtual host setup for the benchmark server: TechEmpower/FrameworkBenchmarks#5980

SunHttp is also suffering - it seems to think that it responds, but wrk doesn't acknowledge requests are complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants