Correctness testing infrastructure

[cuongdo]

As we accumulate SQL optimizations, we should ensure that we're continuing to generate correct results. While sqllogictest helps with that, it is not likely to be sufficient.

TPC-C's correctness checks are one way of doing this.

Ensuring correctness for large analytical queries, such as those found in TPC-H and the analytical parts of CH-benCHmark, is vital to building credibility with customers.

Some possible approaches are:

• Run against a static data set, such as TPC-H. They provide an answer set for easy comparison.
• Compare against the upstream database, such as MySQL in our CH-benCHmark demo setup. This may require pausing transaction load or somehow being able to execute a query as of the same GTID for both MySQL and Materialize.

[rjnn]

I'd say the priority ordering (in increasing order of complexity):

1. Run the static TPC-H queries since have 'known good' answers that we can easily check for.
2. Do the TPC-C correctness views as described in correctness testing using TPC-C checks #486.
3. Use a variety of load generators that we can find, running the queries in MySQL, getting the GTID and then repeating the query in Materialize using as of GTID, and looking for exact matches.
4. Adapt an open source tool like SQLSmith for Materialize, using the same as of GTID approach as above.

[cuongdo]

It seems like items 1 and 2 seem doable and advisable for the PoC period. I'm not as certain about items 3 and 4.

[rjnn]

We should also do item 1, and perhaps add that into the integration tests: we should be able to run dbgen to generate scalefactor-1 CSV files, and define TPC-H views and use that to round-trip test defining views over files and whether the answers computed are correct, and make that an integration test that runs nightly.