Apache (4 and 5) server engine high memory consumption under load

[strohel]

Hi, when doing some benchmarking of various server engines (that also resulted in fixing #141, thx @daviddenton), I've found that apache4 and apache (5) engines' memory consumption rises sharply starting with around 64 concurrent connections. No other engines exhibit this issue.

See the graphs: https://storage.googleapis.com/strohel-pub/bench-http4k-server-engines-apachemem/bench-results.html (Max Memory Usage vs. Connections)

http4k runs in a Docker container with memory limit of 512 MiB, in OpenJDK 11 with -XX:MaxRAMPercentage=75 (allow Java heap up to 75% of assigned memory limit). The container is actually killed on Out-Of-Memory when (extreme) 1024 concurrent connections are fired at it.

http4k version: 3.258.0
code: https://gitlab.com/gooutopensource/locations-kt-http4k/-/tree/kt-http4k-apache4

I suspected this could be caused by the default socket backlog size of 1000, so I've tried with a lowered socket backlog of 100, with no effect on performance or memory consumption (the apache4-q100-* lines in the graphs).

This may not be a bug but rather a property, I'm filling this mainly to consult, also because apache(4) is otherwise the most performant engine.

[daviddenton]

We have now introduced Apache 4, 5 and Netty to the TechEmpower benchmarks. Latest results are here: https://www.techempower.com/benchmarks/#section=test&runid=203ff846-d210-4205-851f-611c5c246c8d

Unfortunately for some reason, Apache 5 is not starting in the test runs - I think this is to do with matching of the canonical hostname in the creation of the server, but can't yet work out how fix it (as it passes the TE build stage).

Netty looks good though... :)

[strohel]

Unfortunately for some reason, Apache 5 is not starting in the test runs - I think this is to do with matching of the canonical hostname in the creation of the server, but can't yet work out how fix it (as it passes the TE build stage).

Maybe I had a similar problem with Apache 5.0 backend locally: it would reply correctly on http://localhost:8080/, but would respond with HTTP 400 + "Not authoritative" in the body on http://127.0.0.1:8080/. I think the culprit is in:

https://github.com/apache/httpcomponents-core/blob/edeb19b1a7490f6753ce118ac7d471a974736be6/httpcore5/src/main/java/org/apache/hc/core5/http/protocol/RequestHandlerRegistry.java#L93-L109

The HttpComponents seem to try to do host-based virtualhosting here... (it seems that every thing called apache needs to do virtualhosting... ;) )

[daviddenton]

I think I've fixed this in the benchmarks by messing with the virtual host setup for the benchmark server: TechEmpower/FrameworkBenchmarks#5980

SunHttp is also suffering - it seems to think that it responds, but wrk doesn't acknowledge requests are complete.