TPC-H benchmarks

[jonkeane]

• Add answer correctness validation
• Add an option to use IPC files as base
• Figure out why codecov is unhappy

Possible scope creeps (that probably aren't worth doing right now, if we're going to separate out and have a separate library for managing sources)

• refactor known_sources to use the generator pattern I used with TPC-H where the download step would live in a function and each known_source would have that function available

[nealrichardson]

You could also do this as:

tpc_h_queries <- list()
tpc_h_queries[[1]] <- function(input_func) ...
tpc_h_queries[[6]] <- function(input_func) ...

Could be more intuitive than stringified number names

[jonkeane]

Ah, yes of course. This is much better

[nealrichardson]

Why not open_dataset()? I wonder if that has different/better properties, even if it is a single file. (Hey we should benchmark that!) It at least shouldn't be memory constrained.

[jonkeane]

I'll add that as an option and see if there are better properties. You're right we probably should always do open_dataset() here even if these are typically one-file datasets.

[nealrichardson]

So we have to read in all tpch_files now? Do these queries share the same dataset, or do we get 22 datasets for 22 queries?

Since each process is only running a single query (right?) it seems like you should be able to only read in the ones that the current query requires

[jonkeane]

That is what this is doing, yeah. Each query uses the same dataset (well dataset of multiple tables). This query only needs one table (lineitem), so we really only need to read that in.

I can make a query to table map so that we only read in the tables that are necessary for each query.

Ultimately, I don't think this will have all that much of an impact, lineitem is the largest table by far, so reading the others in shouldn't have a huge impact even if they aren't used.

Additionally, right now what happens is that each of the files are memory mapped by this, but aren't actually paged into memory until the first run (this is why for arrow the first run in this condition is so much longer than the others). I'm planning to turn memory mapping off so that we really are starting with a truly in-memory table (though we could parameterize that if we wanted to!)

[jonkeane]

Well, it turns out open_dataset("single_file.parquet") is significantly faster than reading the file in (even as an Arrow table) and operating on it. I'm a little bit surprised (especially for the feather files!), but I wonder if the dataset scans of the files are more optimized than a query against a table that is backed by a file?

I'm also pretty surprised that open_dataset(feather_file) is faster than the query engine against the table already resident in memory (the "native" format).

Any thoughts about an explanation for these oddities @bkietz?

With datasets, our performance against parquet files is in line with DuckDB's, though our native query processing is considerably longer.

Turning off memory mapping has the impacts I expected:

• For the native workflow, the first iteration is no longer considerably longer (since the file is already resident in memory before timing starts)
• For the feather and parquet there's mixed results — here reading the file in happens during the benchmark timing.

TPC-H.html.zip

We have a few questions that we need to anser:

• should we keep the read_parquet|feather() along side open_dataset() or just use the dataset processing? I can leave both in and only use one as a default (though if we were to do that, I would change the way they are specified).

• what our defaults should be (and what should be run on every commit in conbench — which should be the same thing though they could technically be different). I propose:

  • scale: 1, 10, (possibly 100 if we can get the ursa machines to generate that much data)
  • engine: arrow
  • format: native, parquet, feather (with the two files being driven by datasets)
  • mem_map: false (only applies to native, this will make the first iteration more in line with the others)

[jonkeane]

@nealrichardson I'm planning to merge this today, unless you have any other things you would like to change or more time to look at it.