Benchmark
The primary objective of this comparison is to evaluate the performance, scalability, and reliability of four API frameworks under consideration. By analyzing key metrics, we aim to provide a data-backed report that will guide the decision-making process for adopting a framework for our next project.
The performance testing is executed using the k6 library from Grafana, a powerful tool that allows for distributed load testing with detailed real-time insights. It is known for its ability to simulate complex scenarios and varied traffic patterns, making it a go-to choice for API testing.
The tests are conducted on a machine equipped with an Intel with 8 cores running at 4.20 GHz and 32GB RAM. This ensures that the tests themselves are not bottlenecked by hardware limitations, providing a fair testing ground for the evaluated frameworks.
Ubuntu 20.04.5 running on Windows Subsystem
Frameworks/HTTP servers to be compared.
- ExpressoTS (Framework)
- Nestjs (Framework)
- Expressjs (HTTP Server)
- Fastify (HTTP Server)
We have chosen several key metrics to provide a comprehensive view of how each framework performs under load, as well as its efficiency and reliability.
-
http_req_duration: The total time taken for a request to complete from the client's perspective. It is the sum ofhttp_req_connecting,http_req_sending,http_req_waiting, andhttp_req_receiving. It measures how long it takes to make an HTTP request, and it's one of the most critical indicators of the API's performance.- Min: Minimum time observed.
- Max: Maximum time observed.
- Average: Arithmetic mean of the observed times.
-
http_req_waiting: The time spent waiting for the server to send a response. This is essentially the server processing time, and it’s the part of the request where your application code will run. -
http_req_connecting: Time spent establishing a TCP connection to the remote host. This generally should be a low value unless there are network issues or the remote host is slow to respond. -
http_req_tls_handshaking: Time spent handshaking the TLS session. This should be a relatively low value, but if it’s too high, you may want to consider optimizing your TLS settings. A dash "-" here usually indicates that the metric is not applicable, perhaps because the connection is not using TLS. -
http_req_sending: Time spent sending data to the remote host. This generally includes sending request headers and any request body. -
http_req_receiving: Time taken to read the response from the remote host. This is the time taken after the first byte is received and until the last byte of the response body is received. -
http_req_blocked: Time spent blocked before initiating the request. This could be time spent waiting for a free TCP connection or time spent doing a DNS lookup. -
iteration_duration: This is the total time taken for one full iteration of the script to be executed, including any setup and teardown functions. It gives you an idea of how long each user simulated by k6 takes to go through the scripted actions.
scenarios: (100.00%) 1 scenario, 1000 max VUs, 1m30s max duration
ExpressoTS
| Metric | Average | Minimum | Median | Maximum | 90th Percentile | 95th Percentile | Count | Rate |
|---|---|---|---|---|---|---|---|---|
| data_received | - | - | - | 14 MB | - | - | - | 236 kB/s |
| data_sent | - | - | - | 4.8 MB | - | - | - | 79 kB/s |
| http_req_blocked | 1.51 ms | 600 ns | 1.4 µs | 1.14 s | 2.8 µs | 4.3 µs | - | - |
| http_req_connecting | 1.48 ms | 0 s | 0 s | 1.14 s | 0 s | 0 s | - | - |
| http_req_duration | 6.20 ms | 137.9 µs | 802.1 µs | 403.08 ms | 5.2 ms | 8.9 ms | 59985 | 986.73/s |
| http_req_receiving | 39.71 µs | 8.1 µs | 34.1 µs | 1.24 ms | 57.4 µs | 69.2 µs | - | - |
| http_req_sending | 62.35 µs | 2.7 µs | 5.4 µs | 92 ms | 17.8 µs | 26.1 µs | - | - |
| http_req_tls_handshaking | 0 s | 0 s | 0 s | 0 s | 0 s | 0 s | - | - |
| http_req_waiting | 6.12 ms | 108.2 µs | 757.79 µs | 402.85 ms | 2.93 ms | 6.25 ms | - | - |
| http_reqs | - | - | - | - | - | - | 59985 | 986.73/s |
| iteration_duration | 1 s | 1 s | 1 s | 2.15 s | 1 s | 1 s | 59985 | 986.73/s |
| vus | - | 395 | - | 1000 | - | - | 395 | - |
| vus_max | - | 1000 | - | 1000 | - | - | 1000 | - |
NestJs
| Metric | Average | Minimum | Median | Maximum | 90th Percentile | 95th Percentile | Count | Rate |
|---|---|---|---|---|---|---|---|---|
| data_received | - | - | - | 14 MB | - | - | - | 231 kB/s |
| data_sent | - | - | - | 4.8 MB | - | - | - | 79 kB/s |
| http_req_blocked | 2.08 ms | 700 ns | 2.2 µs | 1.24 s | 3.5 µs | 4.4 µs | - | - |
| http_req_connecting | 2.05 ms | 0 s | 0 s | 1.23 s | 0 s | 0 s | - | - |
| http_req_duration | 6.88 ms | 172 µs | 895 µs | 506.51 ms | 2.99 ms | 6.32 ms | 59959 | 983.43/s |
| http_req_receiving | 47.09 µs | 9.9 µs | 44.6 µs | 5.07 ms | 69.9 µs | 81.69 µs | - | - |
| http_req_sending | 84.43 µs | 2.8 µs | 7.6 µs | 32.21 ms | 20.6 µs | 27.7 µs | - | - |
| http_req_tls_handshaking | 0 s | 0 s | 0 s | 0 s | 0 s | 0 s | - | - |
| http_req_waiting | 6.74 ms | 145 µs | 839.5 µs | 506.42 ms | 5.15 ms | 6.84 ms | - | - |
| http_reqs | - | - | - | - | - | - | 59959 | 983.43/s |
| iteration_duration | 1 s | 1 s | 1 s | 2.24 s | 1 s | 1 s | 59959 | 983.43/s |
| vus | - | 668 | - | 1000 | - | - | 668 | - |
| vus_max | - | 1000 | - | 1000 | - | - | 1000 | - |
ExpressJs
| Metric | Average | Minimum | Median | Maximum | 90th Percentile | 95th Percentile | Count | Rate |
|---|---|---|---|---|---|---|---|---|
| data_received | - | - | - | - | - | - | 15 MB | 243 kB/s |
| data_sent | - | - | - | - | - | - | 4.8 MB | 79 kB/s |
| http_req_blocked | 1.4ms | 700ns | 1.6µs | 267.82ms | 3µs | 5.3µs | - | - |
| http_req_connecting | 1.34ms | 0s | 0s | 265.39ms | 0s | 0s | - | - |
| http_req_duration | 5.78ms | 120.1µs | 792.7µs | 298.9ms | 5.65ms | 11.34ms | - | - |
| { expected_response:true } | 5.78ms | 120.1µs | 792.7µs | 298.9ms | 5.65ms | 11.34ms | - | - |
| http_req_failed | - | - | - | - | - | - | 0.00% | ✓ 0 ✗ 60000 |
| http_req_receiving | 38.6µs | 7.9µs | 32.5µs | 2ms | 57.5µs | 70.2µs | - | - |
| http_req_sending | 106.53µs | 2.7µs | 5.5µs | 36.14ms | 19.4µs | 27µs | - | - |
| http_req_tls_handshaking | 0s | 0s | 0s | 0s | 0s | 0s | - | - |
| http_req_waiting | 5.64ms | 103µs | 746.3µs | 298.84ms | 5.6ms | 11.3ms | - | - |
| http_reqs | - | - | - | - | - | - | 60000 | 987.586604/s |
| iteration_duration | 1s | 1s | 1s | 1.51s | 1s | 1.01s | - | - |
| iterations | - | - | - | - | - | - | 60000 | 987.586604/s |
| vus | - | - | - | max=1000 | - | - | 247 | min=247 |
| vus_max | - | - | - | max=1000 | - | - | 1000 | min=1000 |
Fastify
| Metric | Average | Minimum | Median | Maximum | 90th Percentile | 95th Percentile | Count | Rate |
|---|---|---|---|---|---|---|---|---|
| data_received | - | - | - | - | - | - | 11 MB | 181 kB/s |
| data_sent | - | - | - | - | - | - | 4.8 MB | 79 kB/s |
| http_req_blocked | 1.39ms | 600ns | 1.6µs | 293.67ms | 3.4µs | 6.4µs | - | - |
| http_req_connecting | 1.36ms | 0s | 0s | 293.38ms | 0s | 0s | - | - |
| http_req_duration | 3.81ms | 37.9µs | 497.9µs | 201.49ms | 2.63ms | 4.66ms | - | - |
| { expected_response:true } | 3.81ms | 37.9µs | 497.9µs | 201.49ms | 2.63ms | 4.66ms | - | - |
| http_req_failed | - | - | - | - | - | - | 0.00% | ✓ 0 ✗ 60000 |
| http_req_receiving | 26.11µs | 5.8µs | 16.6µs | 10.17ms | 47.6µs | 68µs | - | - |
| http_req_sending | 80.41µs | 2.6µs | 6.2µs | 71.73ms | 25.4µs | 49µs | - | - |
| http_req_tls_handshaking | 0s | 0s | 0s | 0s | 0s | 0s | - | - |
| http_req_waiting | 3.71ms | 25.9µs | 465.59µs | 196.53ms | 2.56ms | 4.56ms | - | - |
| http_reqs | - | - | - | - | - | - | 60000 | 990.430788/s |
| iteration_duration | 1s | 1s | 1s | 1.44s | 1s | 1s | - | - |
| iterations | - | - | - | - | - | - | 60000 | 990.430788/s |
| vus | - | - | - | max=1000 | - | - | 1000 | min=1000 |
| vus_max | - | - | - | max=1000 | - | - | 1000 | min=1000 |
| Framework | Average Duration (ms) | Median Duration (µs) | Maximum Duration (ms) | 90th Percentile (ms) | 95th Percentile (ms) | Count | Rate (req/s) |
|---|---|---|---|---|---|---|---|
| Fastify | 3.81 | 497.9 | 201.49 | 2.63 | 4.66 | 60000 | 990.43 |
| ExpressJs | 5.78 | 792.7 | 298.9 | 5.65 | 11.34 | 60000 | 987.59 |
| ExpressoTS | 6.20 | 802.1 | 403.08 | 2.93 | 6.25 | 59985 | 986.73 |
| NestJs | 6.88 | 895 | 506.51 | 5.15 | 6.84 | 59959 | 983.43 |
Request Rate
| Framework | Request Rate (req/s) |
|---|---|
| Fastify | ▮▮▮▮▮▮▮▮▮▮▮ |
| ExpressJs | ▮▮▮▮▮▮▮▮▮▮▮ |
| Nest | ▮▮▮▮▮▮▮▮▮▮ |
| ExpressoTS | ▮▮▮▮▮▮▮▮▮▮ |
Average Request Duration
| Framework | Request Duration |
|---|---|
| Fastify | ▮▮▮▮ |
| ExpressJs | ▮▮▮▮▮▮ |
| ExpressoTS | ▮▮▮▮▮▮▮▮▮ |
| Nest | ▮▮▮▮▮▮▮▮▮▮ |
In our comparative analysis, we observed that ExpressoTS outperforms NestJS in specific metrics, while demonstrating comparable performance in others. It's crucial to note that both frameworks leverage either Expressjs or Fastify under the hood. As such, their performances are inherently interrelated and are likely to exhibit similarities.
However, our intent is not to crown one framework as categorically superior to the other. Each framework comes with its own set of trade-offs, encapsulating both advantages and intrinsic costs. As you integrate more advanced functionalities or abstraction layers to simplify development processes, the performance characteristics may shift. What we aim to highlight is the importance of finding an optimal balance between performance and usability—this equilibrium is what defines an exceptional framework.
Moreover, while performance is undeniably a critical metric, it is by no means the sole consideration in the context of application development. Factors such as development velocity, ease of maintenance, and the range of available features are equally impactful. Therefore, when selecting a framework, it's imperative to undertake a comprehensive evaluation that accounts for both quantitative performance metrics and qualitative attributes like developer experience and ecosystem support.
In conclusion, making an informed choice of a framework involves a nuanced understanding of the unique requirements of your project and how each tool can best meet those needs. It is this multifaceted approach to framework selection that leads to sustainable and efficient application development.
- Benchmark source code can be found here: Benchmark
Happy coding 🎉 🎉 🎉