workload: dataset generated by init tpch leads to "wrong output" errors

[RaduBerinde]

ubuntu@radu-kv-0004:~$ ./workload run tpch --concurrency 100 --ramp 5m
I210424 00:32:58.877910 1 workload/cli/run.go:376  [-] 1  creating load generator...
I210424 00:32:58.878346 1 workload/cli/run.go:407  [-] 2  creating load generator... done (took 436.316µs)
I210424 00:33:02.956306 12 workload/tpch/tpch.go:479  [-] 3  [q1] returned 4 rows after 4.08 seconds
I210424 00:33:03.252129 12 workload/tpch/tpch.go:479  [-] 4  [q2] returned 100 rows after 0.30 seconds
I210424 00:33:05.538787 12 workload/tpch/tpch.go:479  [-] 5  [q3] returned 10 rows after 2.29 seconds
I210424 00:33:06.985734 13 workload/tpch/tpch.go:479  [-] 6  [q1] returned 4 rows after 5.11 seconds
I210424 00:33:07.276092 13 workload/tpch/tpch.go:479  [-] 7  [q2] returned 100 rows after 0.29 seconds
Error: TPCH wrong output [q4] returned wrong number of rows: got 4, expected 5

Build is 09be4f7 plus #64149 (which doesn't change anything in sql).

[RaduBerinde]

To initialize, I used:

./workload init tpch --data-loader=import

[RaduBerinde]

Also saw various other errors.. Maybe the data import is incomplete?

E210424 00:40:30.109586 1 workload/cli/run.go:489  [-] 191  TPCH wrong output [q13] returned wrong number of rows: got 36, expected 42
E210424 00:40:17.268104 1 workload/cli/run.go:489  [-] 180  TPCH wrong output [q9] returned wrong number of rows: got 0, expected 175
E210424 00:41:19.998017 1 workload/cli/run.go:489  [-] 230  TPCH wrong output [q11] returned wrong number of rows: got 951, expected 1048

[yuzefovich]

I think that if you import the data this way, it is somewhat non-deterministic (while respecting the TPCH spec). The expected rows are hard-coded for a particular instance stored in our gs bucket. You can restore the data using RESTORE DATABASE tpch FROM 'gs://cockroach-fixtures/workload/tpch/scalefactor=1/backup'; (we also have a backup there for Scale Factor of 10). Alternatively, you can specify --disable-checks=true option when running the workload.

[yuzefovich]

Note that we do run tpchvec roachtest nighly which does check that the expected results are produced.

[yuzefovich]

I imported the data set locally this way and ran the workload with both vectorize on and off:

yuzefovich@Yahors-MacBook-Pro cockroach % ./bin/workload run tpch --concurrency 1      
I210424 03:20:27.810561 1 workload/cli/run.go:376  [-] 1  creating load generator...
I210424 03:20:27.810902 1 workload/cli/run.go:407  [-] 2  creating load generator... done (took 346µs)
I210424 03:20:40.607076 35 workload/tpch/tpch.go:479  [-] 3  [q1] returned 4 rows after 12.80 seconds
I210424 03:20:40.987376 35 workload/tpch/tpch.go:479  [-] 4  [q2] returned 100 rows after 0.38 seconds
I210424 03:20:45.887129 35 workload/tpch/tpch.go:479  [-] 5  [q3] returned 10 rows after 4.90 seconds
Error: TPCH wrong output [q4] returned wrong number of rows: got 4, expected 5
yuzefovich@Yahors-MacBook-Pro cockroach % ./bin/workload run tpch --concurrency 1 --vectorize=off
I210424 03:20:54.361465 1 workload/cli/run.go:376  [-] 1  creating load generator...
I210424 03:20:54.361561 1 workload/cli/run.go:407  [-] 2  creating load generator... done (took 98µs)
I210424 03:21:07.735069 32 workload/tpch/tpch.go:479  [-] 3  [q1] returned 4 rows after 13.37 seconds
I210424 03:21:08.090386 32 workload/tpch/tpch.go:479  [-] 4  [q2] returned 100 rows after 0.36 seconds
I210424 03:21:13.755335 32 workload/tpch/tpch.go:479  [-] 5  [q3] returned 10 rows after 5.66 seconds
Error: TPCH wrong output [q4] returned wrong number of rows: got 4, expected 5

So I'm quite certain it's the difference in the data set to blame.

[RaduBerinde]

Thanks, But then workload needs to be fixed somehow. Running workload init and workload run is the standard way to run some workload and apparently that workflow is broken for tpch.

[yuzefovich]

Ok, I'll make --disable-checks=true the default option.