Benchmark ParadeDB against 1TB Dataset & Update Benchmarks

[philippemnoel]

What
We are starting to get requests from users with larger databases, sometimes up to 1TB+. Right now, we're only benchmarking ParadeDB and pg_bm25 on a pretty small, ~8GB dataset. I propose that we set up a 1TB benchmarking test, which we could run manually or more rarely than the nightly benchmark runs, to test the system at greater scale.

Here are some relevant links I found:

• https://medium.com/botify-labs/experience-working-with-600-tb-elasticsearch-cluster-b6b5a4fa9127
• https://www.elastic.co/fr/blog/benchmarking-and-sizing-your-elasticsearch-cluster-for-logs-and-metrics
• https://github.com/elastic/elasticsearch-opensearch-benchmark
• https://elasticsearch-benchmarks.elastic.co/

Why
^

How
^

[GPF199541]

Our data use case is 10TB+, and we are looking forward to it very much

[philippemnoel]

Our data use case is 10TB+, and we are looking forward to it very much

Sweet! We’ll hopefully get to this before the end of January. I’ll keep you posted. Could you share a bit more about your use case in the meantime?

[philippemnoel]

@neilyio is currently looking into this. We'll also update our benchmarks for pg_bm25, which are a bit outdated