Comparing with non java technologies

[vsevel]

I am working on an evaluation of different technologies in the context of providing recommendations for our auto-scaling use cases.
I have forked this repo, and added 3 new runtimes: nodejs go rust
I ran the different tests with 1 and 2 cores.

Since I am looking at the auto-scaling use case, the primary metrics I am interested in are (in order):

• density (i.e. throughput / memory)
• memory and time after 1st request
• ideally, linear vertical scalability
• lastly, absolute throughput

all runs are archived, and I maintain a synthesis report:
https://github.com/TechAndOperations/spring-quarkus-perf-comparison/blob/feature/density/results/README.md

Since I am interested in density, I tried squeezing the heap on java workloads. I tried 512, 256, 128, 96, 64, 48. And I would retain the best run for each runtime java. For instance I get the best density for quarkus3-native at 96 Mb, and 64 Mb for quarkus3-virtual. I tried to do that as well on nodejs, but I did not see any change in memory usage.

I compared the ratio you get with 4 cores in your lab, and the ones I get in my environment. It is not too far off, but differences of amplitude appear smaller in my environment. Results quality would benefit from being executed in a more controlled machine setup.

The differences between the runtimes have been documented (mainly regarding observability and orm strategy).

If I put aside weak performances on nodejs and spring4-native, I was impressed with the results of rust and go. In absolute throughput, it is true that quarkus3-virtual beats everybody at 7032 req/s, but it consumes 499 Mb of RSS, when rust consumes just 21 Mb and gets 2322 req/s. go gets a better throughput with more memory, but far below quarkus3 (jvm or graalvm).

As a result, I can get a much higher density on rust (115 tps/Mb) and go (63 tps/Mb) compared to quarkus3 jvm or native (around 22 tps/Mb).
Other interesting metric, I am getting good results on build times on 2 cores: 44 secs on go, 188 secs on rust, when a graalvm compilation takes around 300 secs.

I know quarkus/springboot are seen to be the main competitors to each other. But I would love to see other technologies to be considered as well, specially go that shares some similarities with the JVM (e.g. a GC), compiles extremely fast, and gets 3 times the density of quarkus.

We are in the process of reviewing internally the code for nodejs and go. I know our environment is not ideal, but hopefully the tests still have some value. Do not hesitate to comment on the application code or the methodology.

cc @franz1981

[franz1981]

Just FYI we have now the option to extract if the excess of RSS belong to heap or off heap so to tune it accordingly, see https://bugs.openjdk.org/browse/JDK-8318636

I have still few concerns re different aspects:

• shouldn't all partecipants run vs the same load? That would further enable the most efficient framework to be constrained, in memory (via xmx), as it probably has way more CPU time to run GC and keep RSS low
• the JIT mode GCs are not tuned for low RSS by default, while Rust and go usually are (former is relying on malloc/free, latter has being designed for that, assuming a good generational hp workload), see https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/
• without pmap from https://bugs.openjdk.org/browse/JDK-8318636 I am not sure it's all about heap here as Netty uses off heap allocations in Quarkus and it require specific settings to retain less memory and rely more on malloc/free as Rust does (see OOM regression with Netty 4.2.16 under idle-after-burst load quarkus#55776 )

The very first point is quite important as different runtimes react differently to CPU saturation (GCs are mostly background and exploit batching!); in addition to that, we don't know yet if the CPU saturation too apply equally to all of them since they have different peak tps too.

[franz1981]

I am not sure we have exposed the rate parameter yet from qDup/hyperfoil 🙏
Re the RSS maybe the footprint is not the java heap, so adding -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=5000 (or lower) should help

[vsevel]

it does help indeed:

┌─────────────────┬──────┬────────────┬───────┬─────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│       GC        │ -Xmx │ Throughput │  RSS  │ Density │                                                        GC options                                                         │
├─────────────────┼──────┼────────────┼───────┼─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Parallel + Trim │ 64m  │ 5 949      │ 263.7 │ 22.82   │ -XX:+UseParallelGC -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=5000                                       │
├─────────────────┼──────┼────────────┼───────┼─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Parallel        │ 64m  │ 6 070      │ 306.6 │ 20.87   │ -XX:+UseParallelGC                                                                                                        │
├─────────────────┼──────┼────────────┼───────┼─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤

but just like you said, with a heap of 64 Mb and a RSS of around 300 Mb, there is not much you can gain with GC options. Part of the explanation is probably off-heap.

[franz1981]

If it's coming from C2 compilations too, you can force tiered compilations at level C1 without profiling.
You will probably get 1/2 of peak TPS but much lower RSS.
For Netty I will check, once back from vacation

[vsevel]

FYI I have added a constant rate test:
https://github.com/TechAndOperations/spring-quarkus-perf-comparison/blob/feature/density/results/README.md#constant-rate-open-loop

you can force tiered compilations at level C1 without profiling.

I would rather wait for https://openjdk.org/jeps/8335368 and do some proper training with IT

[franz1981]

Now that you have a fixed tpa test you can:

1. Extract via pidstat steady state the actual CPU usage under load for each runtime
2. Reduce dramatically the heap size till matching p50/p90

[franz1981]

One qq @vsevel did you measure the peak RSS?
Or Better, the RSS over time during steady state and get the peak value or an histogram?
Asking this because rust or go could already have released back the memory to the OS and snapshotting at the end could not be representative of the RSS over time during load.

And last but not least.
if the culprit of RSS is glibc malloc, glib is not affinity friendly so to dedicate less "cores" which will be used to size its arenas, is necessary to set MALLOC_ARENA_MAX 2

[vsevel]

did you measure the peak RSS?

I did not. until now I was using the result of the pmap:
https://github.com/quarkusio/spring-quarkus-perf-comparison/blob/main/scripts/perf-lab/main.yml#L737-L747

I did several improvements this morning.

First I took the time to assess the required heap for the quarkus 3 virtual using the open loop test at 2000 req/s:
https://github.com/TechAndOperations/spring-quarkus-perf-comparison/blob/feature/density/results/README.md#assessing-heap

At 48m there was only a slight degradation. At 32 the app collapses.

I changed the pidstat command to report RSS using -r.
I added MALLOC_ARENA_MAX=2.
I created a density python script to capture the latencies, actual throughput, RSS, CPU.
For RSS and CPU I used the data from pidstat during the load test (excluding warmup and cool down). For RSS I took the max, and for the CPU I averaged the values during the period.
Then I averaged those values across the different iterations.

Here is the result:
https://github.com/TechAndOperations/spring-quarkus-perf-comparison/blob/feature/density/results/README.md#density

It shows all runtimes can keep up with the 2000 req/s steady state. The 2 main differences are:

• RSS is far below for go and rust
• percentiles are much better for quarkus

I am going to try out with quarkus native

[vsevel]

the optimal heap for quarkus3-native is 64m (was 48 for quarkus3-virtual). and I get a density of around 14 req/s/Mb, better than quarkus jvm (8 req/s/Mb), but smaller than go (39) and rust (78).
quarkus3-native shows a degradation on latencies compared to the jvm, but stays much better than go and rust.
do you see anything in this testing methodology or tuning that needs to be improved?
should I try increasing the constant rate? I have seen rust with an "Exceeded session limit", so not sure it is going to like it.
https://github.com/TechAndOperations/spring-quarkus-perf-comparison/blob/feature/density/results/README.md#density

[Sanne]

Hi @vsevel , very interesting I love this idea of a broader comparison. I've had it in mind for a long time that we should compare ourselves with a broader set of languages and frameworks, which is why I've been playing quite a bit with the Techempower benchmarks in the past: it resonated with me that there would be such a variety of highly tuned submissions; alas the benchmarks were also quite unrealistic, and are now no longer maintained. And it was fun.

It's tempting to reuse this particular benchmark here, but the original goal and intent is a bit different and such differences do alas have an impact: it was meant to compare with Spring with a focus on easy migrations, aka we'd disallow any form of tuning as it needed to showcase the ability of Quarkus to perform better in an out of the box configuration. It was also meant to be transparent about the figures we post on the homepage.

I do agree with you that Quarkus should be measured vs Rust and similar stacks - but we need to be careful with the perspectives. As an engineer I do want to compare and get it "as close as possible" but realistically while I like a challenge it will be hard to beat it in terms of minimal size - just the spec requirements of the JVM and some of its frameworks won't make it possible. So Quarkus can hopefully approach its efficiency and maintain a better total balance on ROI because the language has such a fantastic ecosystem and tooling, but this needs to be put in perspective and there's an aspect of subjectivity; we can't objectively measure aspects such as maintainability, availability of enteprise support and integrations, etc.. so there's a risk of such metrics getting misinterpreted or misused. We need to do such comparisons as part of engineering in the open, to keep us improving, but can't claim them to be comprehensive benchmarks.

In particular our frameworks, and the JVM as well, do come out of the box with signicant bias towards large multi-core machines an plenty of memory. I believe we might be able to approach the efficiency of rust - with some work and collaborations within the ecosystem - but only if we allow tuning to happen in such a test, which is not allowed in this benchmark at the moment. We'd probably need to also introduce (and document) several more low level flags, perhaps a profile, to let the build system know that we're aiming for "small" over raw throughput.
I think tuning would be fair in this case because Quarkus and the JVM are very flexible, and are often used in such different scenarios - we can't possibly expect it to perform perfect ad-hoc ergonomics for all scenarios.

Any thoughts on this? I'd like to help - I've recently played with developing a new small, native-image based Quarkus app and had some fun making it more efficient - but to do this I've had to use (and in some cases introduce) several low level flags. We could see if these could be sane enough to be made more generally available, either as an advanced tuning guide, or as some sort of profile system.

But again - this might close the gap with rust somewhat while maintaining access to our most advanced components, but I don't think it will beat it in terms of minimal RSS. It might also need to compromise on some "nice to haves" that our typical user is expecting; for example in my experiments I push many more flags to be build-time-only. I think that's necessary but it's been very unpopular for Java users coming from different platforms.

[edeandrea]

Well said @Sanne . I'll also add some additional info. The other challenge, at least for us Quarkus maintainers, is that we don't have much experience/expertise with these other technologies (node/go/rust/etc).

One of the "guiding principles" of this benchmark is that anyone who may copy code from this repo would be copying code that adheres to Quarkus/Java coding best practices. The code is readable and maintainable. Those of us maintaining this repo simply don't have the experience in those other ecosystems to be able to judge what "good" is.

The other challenge is maintainability. We have a hard enough time keeping up/maintaining this. We simply don't have the bandwidth to maintain versions in technology stacks we are unfamilar with (yes, even with the help of AI). Maintenance isn't just about the code in the repo - its also ensuring that things "work", both when running locally on your laptop as well as in the performance lab. We just don't have the bandwidth to invest in it.

To @Sanne 's point about tuning - we do measure both "out of the box/no tuning allowed" and a "tuned" version, but when tuning, the tuning parameters have to be fair and applied consistently to all so as to continue measuring "apples-to-apples".

Hope that gives a little bit more insights.

[vsevel]

thanks for your answers @Sanne and @edeandrea

I completely understand. I did not have any expectation with the community reusing what I did.
I merely piggybacked on this great project, which provided all the ground base to extend it. it was a great experience.
I know it was not developed with that in mind, but for me it was a learning experience, and I got struck by the speed of native build (specifically go, and rust to some extent) and even more by the RSS of these runtimes.

I am not sure java can or even want to try matching those characteristics, as they have been developed with different goals.
that said, I think this can serve as an inspiration, specifically for quarkus that has done a lot of work in the area of performance. if one could say at some point that quarkus is as efficient as go, this would raise the marketing bar ;-) I do not see rust as a competitor. but go has more similarities with java (starting with a gc), and could be seen as one.

how can you help @Sanne ? I would be honored to get your feedback on what I have been doing, the extra runtimes I added, but more importantly the testing methodology. I got a lot of suggestions from @franz1981 and that was greatly useful and appreciated. If only I could end up with being confident in the results I am getting that would excellent.

we all know that perf tests can quickly tell another complete story with just a few tweaks (this morning I introduced a cost calculation based on azure's data provided by claude, which puts a lot of emphasis on cpu vs mem, and as a result gets the jvm bumped up - I am surprised with this data; I am in the process of verifying cost estimates for cpu and mem in our environment).

the question I have to answer to is: provide recommendations when choosing a stack when doing auto-scaling in terms of efficiency (resource cost, throughput, startup time). you are completely correct in saying that the story does not stop here. things like ecoysystem, maintainability, evolution, dev productivity, trained experts, ... are as important.

to summarize you can help with

• providing recommendations on the methodology,
• and relaying in the community that competition is not just about SB. I found this benchmark https://www.tmdevlab.com/mcp-server-performance-benchmark-v2.html to be inspirational for instance.

[Mentrictraining10]

Thanks for sharing this detailed evaluation. The focus on density (throughput per MB of memory) as the primary metric makes a lot of sense for auto-scaling use cases, where resource efficiency can be more important than absolute peak throughput.

The comparison between Quarkus, Spring Boot, Go, Rust, and Node.js is particularly interesting. The results for Rust (115 tps/MB) and Go (63 tps/MB) are quite compelling compared with the ~22 tps/MB you're seeing from the Quarkus JVM/native workloads. The build-time comparison is also noteworthy, especially Go at 44 seconds and Rust at 188 seconds versus roughly 300 seconds for GraalVM compilation.

I also agree that the absolute numbers should be interpreted with some caution given the differences in the test environments. Running the same workloads on a more controlled and identical machine setup would make the comparison much stronger, particularly for memory consumption, vertical scalability, and the 1-core vs 2-core results.

A few additional things I'd be interested in seeing:

Warm-up behaviour: memory and throughput after the first request versus after the application has stabilised.
Scaling efficiency: throughput increase from 1 → 2 cores, and eventually 4+ cores, expressed as a percentage of ideal linear scaling.
Tail latency: p95/p99 latency could be useful alongside throughput, especially for auto-scaling decisions.
Memory under sustained load: RSS at idle, immediately after the first request, and after a longer sustained workload.
Startup time: particularly relevant if the auto-scaling strategy involves rapidly creating new instances.
Concurrency sensitivity: whether the density advantage of Go/Rust remains consistent as concurrency increases.
GC behaviour: especially for Go versus JVM-based runtimes, since GC overhead could become more visible under memory pressure.
Equivalent application behaviour: ensuring that ORM, observability, serialization, connection pooling, and other runtime-specific choices aren't unintentionally favouring one implementation over another.

The heap-sizing results on Java are also useful. The fact that reducing the heap doesn't materially change Node.js memory consumption suggests that the memory footprint there is being driven by factors other than the configured heap, so it would be worth looking at the runtime baseline/native allocations as well.

Overall, I think the results are valuable even if the environment isn't perfectly controlled. They provide a useful direction for the evaluation, and the Go/Rust results are strong enough to justify considering them alongside the traditional Spring Boot vs. Quarkus comparison rather than treating those two as the only candidates.

I'd definitely be interested in seeing the results after the Node.js and Go implementations have been reviewed and the benchmark is repeated on a controlled environment.