Benchmarks differences in keynote-2

[brandonpollack23]

I just was taking a look and found some things im curious about, would you mind explaining why some of these differences were done (different clients and protocols mainly, as well as a lack of warmup on other clients).

https://news.ycombinator.com/item?id=47148920

[Centril]

Hey Brandon,

First, a small note about warmup. Ripping the warmup period out makes no difference for SpacetimeDB's TPS numbers, so that can simply be removed from he benchmark without affecting the results, so we'll do that.

I cannot speak to the benchmarks for the other databases, but I did write the Rust benchmark client, so I can give some insights into that client. I wrote this client to measure the maximum TPS throughput of the host alone, without care for the client's performance. While optimizing the datastore of SpacetimeDB and looking at flamegraphs,
using the typescript client and our higher level APIs, the host was getting bottlenecked on the client using too many resources and not sending requests fast enough.
So I wrote a raw client to do the bare minimum work necessary to talk to the host so that the host's performance could be measured and improved.

Beyond my notes here, we're going to write a proper technical writeup addressing the concerns you've highlighted and those discussed elsewhere as well.

[brandonpollack23]

Thanks so much for your prompt reply. I realized after some things I said weren't quite right so I'm going to add an addendum.

First off setting sqlite with an invalid sync param will likely turn it off according to my genie, and also one thing worth noting that wasn't immediately clear to me is even though sqlite and the others have this extra rpc hop, that is exactly the thing spacetime doesn't have since everything is a stored procedure in the db with wasm. What isn't quite fair is that you use HTTP for those with JSON etc and tons of overhead but Spacetime gets to use a socket in your benchmark.

I also am curious if you compare to other direct backends with the extra server hop (since that's fair) but using a socket connection to the server (which is what a game would do) as well as using NodeJS with SpacetimeDB. Looks like it still uses websockets.

Basically we're comparing two different use cases RPC vs REST which are different so it'd be cool to see apples to apples comparisons :)