Issues with the benchmark results

[puzpuzpuz]

The benchmark results feel more and more like a mess due to a number of issues in the way the benchmark is organized and how the results are shown.

1. As the title says, Cold Run results make no sense with in-memory and remote storage, so ideally they should not be shown on the generated html page.

2. The benchmark rules mention the following:

run.sh: a loop for running the queries; every query is run three times; if it's a database with local on-disk storage, the first query should be run after dropping the page cache;

This rule is not applied to some of the contestants that run on the local disk like Arc which produces wrong results in Combined and Cold Run categories. For instance, Arc uses DuckDB as its query engine: it outperforms DuckDB+parquet in Cold Run by 3.08x, but it's slower than DuckDB in Hot Run by 2.7x - that's quite puzzling. More than that, many if the Arc's Cold Run query execution times are worse than the corresponding Hot Run times.

3. The benchmark rules should specify the disk/volume type and its settings. The machine field used to have gp2 500GB in the text - now it doesn't, so it looks like contestants can use any volume type, e.g. io2, making the Cold Run results questionable.

[puzpuzpuz]

@alexey-milovidov is it on purpose that the benchmark turned into цирк с конями? It used to be a decent, trustworthy benchmark, but that's no longer the case.

[alexey-milovidov]

As the title says, Cold Run results make no sense with in-memory and remote storage, so ideally they should not be shown on the generated html page.

Yes, it will make sense to hide in-memory systems from Cold and Combined results.

This rule is not applied to some of the contestants that run on the local disk like Arc which produces wrong results in Combined and Cold Run categories. For instance, Arc uses DuckDB as its query engine: it outperforms DuckDB+parquet in Cold Run by 3.08x, but it's slower than DuckDB in Hot Run by 2.7x - that's quite puzzling. More than that, many if the Arc's Cold Run query execution times are worse than the corresponding Hot Run times.

Did you find an error in Arc's benchmark scripts? They use a really big / complicated Python script, but at the first glance everything was alright. Let's figure it out.

The benchmark rules should specify the disk/volume type and its settings. The machine field used to have gp2 500GB in the text - now it doesn't, so it looks like contestants can use any volume type, e.g. io2, making the Cold Run results questionable.

This is a wrong accusation. Here you see how the benchmark is run: https://github.com/ClickHouse/ClickBench/blob/main/run-benchmark.sh. The runs are automated to put every contestant in a fair position (for self-hosted entries). As far as I know, none of the systems used io2 disks. No reason to share speculations.

is it on purpose that the benchmark turned into цирк с конями?

There is another phrase, "плохому танцору яйца мешают".

[puzpuzpuz]

Yes, it will make sense to hide in-memory systems from Cold and Combined results.

To me, the same should be done for Cloud systems as Cold/Hot run is undefined for those.

Did you find an error in Arc's benchmark scripts? They use a really big / complicated Python script, but at the first glance everything was alright. Let's figure it out.

I saw no traces of page cache flush in their run.sh and the results suggest that it does not occur between the query runs like it's done for other databases.

This is a wrong accusation. Here you see how the benchmark is run: https://github.com/ClickHouse/ClickBench/blob/main/run-benchmark.sh. The runs are automated to put every contestant in a fair position (for self-hosted entries). As far as I know, none of the systems used io2 disks. No reason to share speculations.

I didn't see the script. In our submissions, I was using results measured on the instance types provisioned with 500GB gp2. The only place where the 500GB gp2 in the readme is mentioned in the Rules and Contribution section is here:

By default, all tests are run on c6a.4xlarge VM in AWS with 500 GB gp2.

This phrase makes an impression that other instance types may use arbitrary volume. Would be great to make this part of the rules more clear.

There is another phrase, "плохому танцору яйца мешают".

Половые органы тут не причем. Речь скорее о том, что части танцоров привязали к ногам пудовые гири. Впрочем, лучшие в данном танцевальном марафоне, с гирями или без, это DuckDB и CedarDB - тут трудно спорить.

Если без шуток, раньше бенчмарк производил лучшее впечатление и был более удобоварим для рядового пользователя СУБД и это не только мое мнение. Здесь я попытался изложить проблемы, которые я вижу - что делать с этим фидбеком и делать ли что-то вообще, решать вам.

[xe-nvdk]

@puzpuzpuz Alexey itself tested this in all the systems that they have, and I don't think that the storage was different there. In our test, we also comply with using GP2 500GB, we don't have any interest of cheating in any way. Test can't be reproduced cause before the launch of the first release, we normalized our api endpoints and stopped to work.

Is quite interesting read your jokes in russian. I had you guys (QuestDB team) as a more serious people, but keep the good mood ;)

[puzpuzpuz]

I had you guys (QuestDB team) as a more serious people, but keep the good mood ;)

@xe-nvdk
No need to mention QuestDB team, I created this issue as an individual who's keen to learn optimizations met in OSS databases and compare their impact on performance, so, I guess, you should blame me. The issue is not focused on Arc: I simply listed the problems accumulated in the benchmark for the past few years, as I see them, and mentioned Arc as a recent example where I didn't find OS page cache flush before each query iteration, a step which is important to reproduce the scenario. Keeping the page cache around is a critical factor since 500GB gp2 is limited with 250MB/s and 3000 IOPS in burst mode, so if the query needs to read a "heavy" varchar column, the volume immediately appears to the bottleneck. This effect is more pronounced on larger boxes like c6a.metal since it has plenty of RAM to keep the data files in the page cache and, in general, is more prone to contention when warming up the pages (when handling major page faults).

When benchmarking various databases, it's important, yet challenging, to put them in the same conditions. This is the goal of any decent benchmark unless the real goal is benchmarketing. Other than that, it appears that this bug in scripts was present in a few databases (#659), not only Arc. To me, it's good to have all the contestants aligned. Needless to say, that I'm sorry if my GH issue led to any drama, that wasn't the goal although Alexey and I had a fruitful eastern-European conversation.