Continuous Benchmarking

[BohuTANG]

Summary

DuckDB's benchmarking are very professional here https://duckdb.org/benchmarks/index.html, thanks to DuckDB for providing such a good testing style :D

Follow the style, databend can also integrate tests:

• Benchmark runner #3263
• Provide the hardware(container) for the runner, this can be based on our perf test
• Run the runner for each nightly version and collect the results to a database(DuckDB using itself for this result stored)
• Show the results on the website for each nightly

References:

1. DuckDB website scripts
2. DuckDB benchmark runner

[ZhiHanZ]

Possible settings:

• 2 workers (4vCPU + 4Gi RAM) run on cloud to benchmark on a medium size dataset (ontime small is fine for now)
  Need to test the following spec

1. test the performance on S3 storage type (COPY， INSERT， aggregation)
2. test the performance on local disk (INSERT, aggregation)

[sundy-li]

Rust benchmark graph: https://perf.rust-lang.org/

[PsiACE]

https://github.com/datafuselabs/databend-perf/ contains numbers and ontime benchmarks, run on a XXXLarge machine, and collects performance data for nightly releases.

• Provide the hardware(container) for the runner, this can be based on our perf test
• Run the runner for each nightly version and collect the results to a database

[ZhiHanZ]

I think it could be closed? @BohuTANG