Most of the best-performing frameworks don't survive temporary db connectivity loss

[itrofimow]

TLDR: in the current top20 only 7 implementations survive a temporary database connectivity loss, other either hang, crash, or get stuck with http500.

---

I decided to become a QA engineer for a day and give frameworks a touch of real-world scenarios, to see how they would react.
I did this

1. Launch the framework
2. Poke it a bit to warm it up
3. Restart the database
4. Poke the framework some more

for every framework in the top20 of the most recent continuous run.

The results i've got:

Framework	Outcome
may-minihttp	hangs
xitca-web	✔️ (#8781)
just	hangs
vertx	stuck with 500/5021
h20	✔️
redkale	✔️
officefloor	hangs
drogon	✔️
viz	stuck with 500/5021
axum	stuck with 500/5021
jobby	hangs
actix	stuck with 500/5021
mormot	✔️
ntex	hangs
vertx-web	stuck with 500/5021
wizzadro_http	✔️
salvo	crashes
aspnetcore	✔️
lithium	✔️
inverno	hangs

---

Some of the rules in the spec are a bit vague, intentionally i believe, and Rule 1. of General Test Requirements is definitely one of such rules, but I would be very surprised to learn that the behavior observed qualifies as a "production-grade" in anyone's book.

Having a reliable way to recover from potential errors is likely to be not free performance-wise.
Some of the frameworks above come with configurations that behave reasonably in the scenario given,
but their reliability cost is not just "not free", it's staggering:
Axum, for example, which hits a very impressive 550K RPS and is ranked 9 in the "Single Query" test,
has a configuration that survives a database restart.
That configurations drops its performance to 90K RPS (x6 degradation²) and rank 262, even below MongoDB-based implementation for the test.

---

Considering the above, from a perspective of a web-developer who tries to choose a right tool for the job I personally find the performance figures shown misleading.

Curious to hear the community thoughts on the matter.

Footnotes

1. By 502 I mean "closes the client connection right away" ↩ ↩² ↩³ ↩⁴ ↩⁵

2. I think it's not configured optimally, but still ↩

[itrofimow]

I assume you've got your browser tabs mixed up, because otherwise I have not a slightest idea about what orm/rust/redis-question you @zheby are talking about.

[itrofimow]

cc @Xudong-Huang @fakeshadow @billywhizz @vietj @sagenschneider @fundon @jplatte @jknack @ecruz-te @fafhrd91 @ShreckYe @chrislearn @jkuhn1 as the most recent committers into the corresponding implementations.

Do you find my concerns justified?

[jplatte]

Hey, I do think that if things don't recover from temporary DB issues, that's pretty weird / probably shouldn't fly for the benchmarks.

Re. axum, the one I contributed to, I can say that the code has some issues (though none that I know would certainly degrade performance) but I haven't had that much motivation to take the time required to fix them. Which configurations are you talking about as the slow and fast one?

[itrofimow]

Hi!

This is actually a common theme for every (as far as I can tell) rust implementation: employing tokio_postgres directly is fast and doesn't recover (https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/axum/src/database_pg.rs#L41), deadpool-based implementations seem to be reliable yet slow (https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/axum/src/database_pg_pool.rs#L43)

[jplatte]

I wouldn't mind at all having the pg impls removed, personally x)
Interesting that sqlx is significantly slower than deadpool again, wouldn't have expected that. Though a lot might have happened since the 2023 results, anyways 🤷🏼

[fafhrd91]

if techempower team say recovery is require, should easy to add. doubt it would change results on measurable amount

[itrofimow]

@NateBrady23 could you please clarify the rules here?

[NateBrady23]

Thanks for doing that QA @itrofimow

I agree with everything said here. I would expect a production-grade framework to be able to recover from a temporary db connectivity loss. However, I also agree that it probably won't alter the results.

Still, I'd like to see the frameworks here updated, though we probably wouldn't be adding this into the test suite anytime soon.

I'll leave this issue open for now. If anyone wanders here, it probably doesn't look good if your framework implementation doesn't recover in such a way.

[fakeshadow]

Mandatory reconnectivity is a good rule to add. Though it's unlikely to affect perf unless it's a hot path of your system.

[bhauer]

Some of the rules in the spec are a bit vague, intentionally i believe, and Rule 1. of General Test Requirements is definitely one of such rules, but I would be very surprised to learn that the behavior observed qualifies as a "production-grade" in anyone's book.

I am the original author of rule 1 of the general test requirements. You are right that it is intentionally vague on the specifics of production grade. My thinking at the time was that if there were specific matters, like the one you raise, the community could discuss them in precisely this manner. So thank you for raising the concern!

Our deference to each framework's definition of "production grade" grants us some luxury to avoid dictating the right or wrong ways to build web applications. For example, I would not want to reject a framework or platform whose developers stated something akin to, "It's normal to restart the application server in production if the database server restarts." While I personally would disagree with such a concession, meanwhile, I would continue not using foreign keys in databases and another developer might consider that reckless. I suspect we can all think of weird stuff we've seen being considered production-grade by other developers.

My inclination would be to allow for the current situation but re-emphasize a longer-term goal of the project to offer more detailed implementation notes for visitors on the results web site. Being able to understand that a given framework provides database connection resilience (or does not) with relative ease would be worthwhile.

In other words: don't forbid this arguably risky optimization (or lack of functionality), but call attention to it, and by so doing, nudge the developers to either explain how they recommend handling database outages in a production environment, present an argument about why it's not relevant in their ecosystem, or accept that consumers of the results data will be made aware of the concession.

[itrofimow]

I would expect a production-grade framework to be able to recover from a temporary db connectivity loss.

Thanks for clarification.

However, I also agree that it probably won't alter the results.

About that: pipelining unrelated queries over a relatively small number of connections seems to be instrumental to squeeze as much performance from PostgreSQL as possible in these benchmarks, however most widely used tokio_postgres simply doesn't have a "reconnect" interface, and the author himself recommends to pool the connections.

Pooling connections takes away the possibility to pipeline unrelated queries, so one needs to increase the number of connections to achieve the same throughput.
Increasing the number of connections affects the database badly, so one might want to implement some kind of complicated synchronization in user-code to keep the connections count small yet fault-tolerant.
Implementing this synchronization in user-code IMO is questionable from a "showcase what the framework can do" point of view.

Curious to learn how folks would overcome this limitation, will see.

P.S. This issue (the github issue I mean, #8790) is not specifically about Rust implementations, it just so happens that they all fail the QA-test for the exact same reason.

edit: typos
edit2: wording

[itrofimow]

Given you interpretation, should I rename the issue to "Most of the best-performing frameworks don't survive temporary db connectivity loss", so it doesn't come as strongly worded w.r.t. the rules? @bhauer

[bhauer]

@itrofimow Up to you! That title revision might be more clear about the specific concern, but I don't think the title matters very much.

[ShreckYe]

Hi @itrofimow, I mainly adapted the Vert.x benchmarks into the Kotlin language. Let me help you cc @vietj, @franz1981, and @pmlopes who are the core developers for this.

[jkuhn1]

Hi @itrofimow

I do agree that production-grade applications should be resilient to DB failure or restart. Now looking at rule 1 (https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview), as you mentioned it is not really obvious that this is required for the performance test (as you mentioned "should" is not "must"). If we want to go on that path, I think it might be good to clarify this, we can keep "should" to be inclusive and yet clearly state that it must be resilient to DB connection loss but this raises some other concerns: TFB must then include a test for this, if we test this what else can we test to make sure application is resilient? (client connection loss? client sending bad HTTP requests...) This can easily lead to test frameworks resiliency and no longer only pure performance. Don't get me wrong this would be a good thing but it would also widen the scope.

Inverno (and I guess many other Java based frameworks) relies on Vert.x SQL client (which is really good). The framework is still under heavy development and I wouldn't use it yet for mission critical applications (and above issue just proved it) but my goal is mission critical production grade so I'll take some time to test this scenario and see if something can be done on my side otherwise I guess @vietj would need to provide some fix or advice (please let me know if I can help).