Use tidb_query_vec_executors instead of normal executors #8173

Renkai · 2020-07-01T09:30:25Z

What problem does this PR solve?

Issue Number: #7039

Problem Summary: migrate some normal executors to vector executors.

Related changes

Nope

Check List

Tests

Unit test
Integration test

Side effects

Nope

Release note

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai · 2020-07-02T07:20:11Z

@zhongzc @iosmanthus @breeswish PTAL

breezewish · 2020-07-03T15:08:12Z

src/coprocessor/statistics/analyze.rs

+
+        let mut is_drained = false;
+        let mut batch_size = BATCH_INITIAL_SIZE;
+        while !is_drained {


I guess there is no need to follow a grow strategy for analyze

I guess there is no need to follow a grow strategy for analyze

So is there a suitable constant for batch_size?

breezewish · 2020-07-03T15:09:39Z

src/coprocessor/statistics/analyze.rs

+                for col in batch.physical_columns.as_slice() {
+                    let buffer_vec = col.raw();
+                    let data = buffer_vec.index(logical_row);
+                    bytes.extend_from_slice(data);


Looks like the behaviour is scanning data from row, convert to column, and here convert it back to row?

Looks like the behaviour is scanning data from row, convert to column, and here convert it back to row?

I think a more suitable description is scanner scanned a matrix, then here get every row from matrix.

Renkai · 2020-07-07T15:35:36Z

@sre-bot /run-all-tests

Renkai · 2020-07-07T15:35:46Z

@sre-bot /release make dist_release

Renkai · 2020-07-07T15:35:56Z

@sre-bot /bench +sysbench +tpcc +tpch

Renkai · 2020-07-08T02:03:11Z

@sre-bot /run-all-tests

Renkai · 2020-07-08T02:46:42Z

@ti-srebot bot /run-all-tests

Renkai · 2020-07-08T02:46:53Z

@ti-srebot /release make dist_release

ti-srebot · 2020-07-08T02:46:54Z

/release make dist_release

Renkai · 2020-07-08T02:47:11Z

@ti-srebot /bench +sysbench +tpcc +tpch

ti-srebot · 2020-07-08T02:47:13Z

/bench +sysbench +tpcc +tpch

sre-bot · 2020-07-08T03:09:13Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/14693554e953a6286667cefdb8dd79261854cce6/centos7/tikv-server.tar.gz

sre-bot · 2020-07-08T03:51:48Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 5f983b860ff77ae9eeb6115404ac82b3e3cfe96d
--- tikv: 4a75902f266fbbc064f0c19a2a681cfe66511bc3
+++ tikv: 14693554e953a6286667cefdb8dd79261854cce6
pd: 9f1a55c0e36dba3ec35447d16d58dfd9d6becb90
================================================================================
oltp_update_index:
    * QPS: 5289.07 ± 5.95% (std=237.93) delta: 4.07% (p=0.671)
    * Latency p50: 26.70 ± 8.44% (std=1.60) delta: 2.84%
    * Latency p99: 15.27 ± 0.00% (std=0.00) delta: -0.00%
            
oltp_insert:
    * QPS: 4636.81 ± 7.22% (std=286.86) delta: -6.21% (p=0.529)
    * Latency p50: 27.61 ± 7.75% (std=1.74) delta: 2.81%
    * Latency p99: 12.75 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_read_write:
    * QPS: 9319.09 ± 19.31% (std=1093.05) delta: 11.66% (p=0.500)
    * Latency p50: 261.62 ± 4.86% (std=9.90) delta: -15.76%
    * Latency p99: 152.13 ± 2.41% (std=2.60) delta: -98.46%
            
oltp_point_select:
    * QPS: 113032.25 ± 0.22% (std=179.04) delta: 0.26% (p=0.326)
    * Latency p50: 1.13 ± 0.00% (std=0.00) delta: -0.44%
    * Latency p99: 4.25 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_update_non_index:
    * QPS: 5650.81 ± 3.31% (std=152.25) delta: -13.82% (p=0.347)
    * Latency p50: 22.62 ± 3.57% (std=0.65) delta: 12.77%
    * Latency p99: 12.98 ± 0.00% (std=0.00) delta: 0.00%

sre-bot · 2020-07-08T03:56:49Z

Benchmark Report

Run TPC-C Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 5f983b860ff77ae9eeb6115404ac82b3e3cfe96d
--- tikv: 4a75902f266fbbc064f0c19a2a681cfe66511bc3
+++ tikv: 14693554e953a6286667cefdb8dd79261854cce6
pd: 9f1a55c0e36dba3ec35447d16d58dfd9d6becb90
================================================================================
Measured tpmC (NewOrders): 224.58 ± 34.43% (std=47.22), delta: -3.39% (p=0.833)

sre-bot · 2020-07-08T05:26:31Z

Benchmark Report

Run TPC-H 10G Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 5f983b860ff77ae9eeb6115404ac82b3e3cfe96d
--- tikv: 4a75902f266fbbc064f0c19a2a681cfe66511bc3
+++ tikv: 14693554e953a6286667cefdb8dd79261854cce6
pd: 9f1a55c0e36dba3ec35447d16d58dfd9d6becb90
================================================================================
01.sql duration: 26344.89 ms ± 19.34% (std=3065.74) delta: -7.88% (p=0.822)
02.sql duration: 2554.56 ms ± 282.62% (std=7567.80) delta: 5.77% (p=0.931)
03.sql duration: 7406.89 ms ± 60.99% (std=8954.27) delta: 1.20% (p=0.900)
04.sql duration: 3940.23 ms ± 112.39% (std=9633.16) delta: 4.80% (p=0.937)
05.sql duration: 7703.84 ms ± 74.73% (std=8375.75) delta: -1.86% (p=0.267)
06.sql duration: 4726.95 ms ± 104.34% (std=11799.81) delta: 6.57% (p=0.464)
07.sql duration: 6955.64 ms ± 63.49% (std=9518.08) delta: 4.38% (p=0.496)
08.sql duration: 7514.47 ms ± 59.96% (std=7204.89) delta: 1.05% (p=0.291)
09.sql duration: 28229.91 ms ± 13.41% (std=2089.12) delta: 5.65% (p=0.465)
10.sql duration: 4916.70 ms ± 79.92% (std=9835.16) delta: -0.10% (p=0.926)
11.sql duration: 2038.38 ms ± 1.28% (std=15.34) delta: 1.72% (p=0.941)
12.sql duration: 6660.12 ms ± 63.83% (std=9288.09) delta: 4.83% (p=0.548)
13.sql duration: 6628.30 ms ± 62.93% (std=8262.63) delta: -0.60% (p=0.569)
14.sql duration: 5627.59 ms ± 79.88% (std=9916.37) delta: 2.65% (p=0.570)
15.sql duration: 24760.70 ms ± 16.48% (std=2979.07) delta: -7.19% (p=0.763)
16.sql duration: 2767.76 ms ± 1.06% (std=19.27) delta: 1.40% (p=0.887)
17.sql duration: 24177.25 ms ± 20.74% (std=3748.66) delta: -17.19% (p=0.709)
18.sql duration: 31290.64 ms ± 2.30% (std=364.37) delta: 2.63% (p=0.180)
19.sql duration: 7090.92 ms ± 66.69% (std=8005.47) delta: 3.05% (p=0.414)
20.sql duration: 5434.19 ms ± 69.34% (std=9497.88) delta: -0.25% (p=0.669)
21.sql duration: 28618.48 ms ± 8.18% (std=1360.76) delta: 27.03% (p=0.704)
22.sql duration: 4163.15 ms ± 105.38% (std=8630.79) delta: 0.33% (p=0.480)

Renkai · 2020-07-08T11:50:09Z

@ti-srebot /bench +sysbench

ti-srebot · 2020-07-08T11:50:11Z

/bench +sysbench

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai · 2020-07-08T11:57:51Z

@ti-srebot /release make dist_release

Renkai · 2020-07-08T11:58:02Z

@ti-srebot /bench +sysbench +tpcc +tpch

ti-srebot · 2020-07-08T11:58:05Z

/bench +sysbench +tpcc +tpch

sre-bot · 2020-07-08T13:17:47Z

Benchmark Report

Run TPC-C Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: a99fdc098cb3bccc63ca74dad3d682b6f80a0fbd
--- tikv: 4a75902f266fbbc064f0c19a2a681cfe66511bc3
+++ tikv: 4d7ec0dc2041b2b5cf50e9b97d359c9d5ae92b67
pd: 9f1a55c0e36dba3ec35447d16d58dfd9d6becb90
================================================================================
Measured tpmC (NewOrders): 245.72 ± 9.85% (std=16.47), delta: -13.48% (p=0.435)

sre-bot · 2020-07-08T13:26:26Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: a99fdc098cb3bccc63ca74dad3d682b6f80a0fbd
--- tikv: 4a75902f266fbbc064f0c19a2a681cfe66511bc3
+++ tikv: 4d7ec0dc2041b2b5cf50e9b97d359c9d5ae92b67
pd: 9f1a55c0e36dba3ec35447d16d58dfd9d6becb90
================================================================================
oltp_update_index:
    * QPS: 4612.81 ± 20.87% (std=633.12) delta: -6.88% (p=0.460)
    * Latency p50: 28.23 ± 18.68% (std=3.69) delta: 7.57%
    * Latency p99: 15.00 ± 0.00% (std=0.00) delta: 1.83%
            
oltp_insert:
    * QPS: 4851.42 ± 0.50% (std=19.88) delta: 7.15% (p=0.411)
    * Latency p50: 27.48 ± 5.62% (std=1.23) delta: -3.46%
    * Latency p99: 12.87 ± 0.89% (std=0.12) delta: 0.90%
            
oltp_read_write:
    * QPS: 9531.92 ± 5.74% (std=386.69) delta: 15.69% (p=0.230)
    * Latency p50: 272.55 ± 5.95% (std=11.46) delta: -13.17%
    * Latency p99: 154.01 ± 4.16% (std=4.67) delta: -98.50%
            
oltp_point_select:
    * QPS: 110425.02 ± 0.71% (std=538.75) delta: -0.37% (p=0.908)
    * Latency p50: 1.16 ± 0.00% (std=0.00) delta: 0.00%
    * Latency p99: 4.41 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_update_non_index:
    * QPS: 6083.40 ± 21.35% (std=835.26) delta: 0.01% (p=1.000)
    * Latency p50: 20.99 ± 2.92% (std=0.44) delta: -1.30%
    * Latency p99: 12.90 ± 1.19% (std=0.11) delta: -0.59%

breezewish · 2020-07-13T05:14:12Z

/bench +sysbench +tpcc +tpch

mahjonp · 2020-07-13T05:17:51Z

Seems that tpch q21 always hangs when benching on the master branch..

sre-bot · 2020-07-13T06:14:27Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 165bdfdec842cf7aac4cb6f80b55db407135ca3b
--- tikv: 1709de63b3b66f474ff757133a8a1076cf77f196
+++ tikv: 1e96bc2a366b517ef3ee4539c0e1c1a59ea70cf8
pd: cb76a79a75ad0d7d2e7f1c27c07d7c2e6a66cac6
================================================================================
oltp_update_index:
    * QPS: 18331.07 ± 0.41% (std=46.49) delta: -0.53% (p=0.185)
    * Latency p50: 6.98 ± 0.43% (std=0.02) delta: 0.50%
    * Latency p99: 16.12 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_insert:
    * QPS: 18937.63 ± 0.07% (std=9.50) delta: -0.27% (p=0.642)
    * Latency p50: 6.75 ± 0.10% (std=0.00) delta: 0.24%
    * Latency p99: 13.22 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_read_write:
    * QPS: 36426.83 ± 0.04% (std=10.00) delta: -0.39% (p=0.032)
    * Latency p50: 71.07 ± 0.05% (std=0.02) delta: 0.38%
    * Latency p99: 125.52 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_point_select:
    * QPS: 112807.95 ± 0.28% (std=204.67) delta: -0.30% (p=0.208)
    * Latency p50: 1.13 ± 0.00% (std=0.00) delta: 0.00%
    * Latency p99: 4.25 ± 0.00% (std=0.00) delta: 0.00%
            
oltp_update_non_index:
    * QPS: 23983.44 ± 0.03% (std=5.63) delta: -0.25% (p=0.051)
    * Latency p50: 5.33 ± 0.00% (std=0.00) delta: 0.14%
    * Latency p99: 13.58 ± 0.88% (std=0.12) delta: 0.89%

sre-bot · 2020-07-13T06:23:22Z

Benchmark Report

Run TPC-C Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 165bdfdec842cf7aac4cb6f80b55db407135ca3b
--- tikv: 1709de63b3b66f474ff757133a8a1076cf77f196
+++ tikv: 1e96bc2a366b517ef3ee4539c0e1c1a59ea70cf8
pd: cb76a79a75ad0d7d2e7f1c27c07d7c2e6a66cac6
================================================================================
Measured tpmC (NewOrders): 3920.11 ± 19.57% (std=594.65), delta: 6.89% (p=0.588)

codecov · 2020-07-13T06:34:13Z

Codecov Report

Merging #8173 into master will decrease coverage by 0.04%.
The diff coverage is 85.48%.

@@            Coverage Diff             @@
##           master    #8173      +/-   ##
==========================================
- Coverage   84.77%   84.72%   -0.05%     
==========================================
  Files         553      553              
  Lines      137646   137911     +265     
==========================================
+ Hits       116688   116851     +163     
- Misses      20958    21060     +102

Impacted Files	Coverage Δ
cmd/src/server.rs	`0.00% <0.00%> (ø)`
components/raftstore/src/coprocessor/mod.rs	`49.20% <ø> (ø)`
...omponents/raftstore/src/store/worker/raftlog_gc.rs	`90.43% <ø> (-0.17%)`	⬇️
components/test_raftstore/src/router.rs	`55.55% <0.00%> (-5.06%)`	⬇️
components/tidb_query_codegen/src/rpn_function.rs	`88.71% <ø> (ø)`
components/tidb_query_vec_aggr/src/summable.rs	`35.71% <ø> (ø)`
components/tidb_query_vec_expr/src/impl_compare.rs	`88.00% <ø> (ø)`
components/tidb_query_vec_expr/src/impl_math.rs	`87.64% <ø> (ø)`
components/tidb_query_vec_expr/src/impl_op.rs	`83.43% <0.00%> (ø)`
src/server/server.rs	`86.82% <0.00%> (-1.37%)`	⬇️
... and 98 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a4a0c9...1e96bc2. Read the comment docs.

sre-bot · 2020-07-13T06:34:40Z

Benchmark Report

Run TPC-H 10G Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 165bdfdec842cf7aac4cb6f80b55db407135ca3b
--- tikv: 1709de63b3b66f474ff757133a8a1076cf77f196
+++ tikv: 1e96bc2a366b517ef3ee4539c0e1c1a59ea70cf8
pd: cb76a79a75ad0d7d2e7f1c27c07d7c2e6a66cac6
================================================================================
01.sql duration: 10522.54 ms ± 0.50% (std=33.14) delta: 2.11% (p=0.000)
02.sql duration: 2343.65 ms ± 3.74% (std=55.28) delta: -1.07% (p=0.518)
03.sql duration: 7349.36 ms ± 2.63% (std=126.61) delta: 1.11% (p=0.086)
04.sql duration: 3488.89 ms ± 0.96% (std=19.12) delta: 3.85% (p=0.002)
05.sql duration: 7932.38 ms ± 0.50% (std=19.13) delta: 1.57% (p=0.892)
06.sql duration: 4760.21 ms ± 1.88% (std=52.92) delta: 3.19% (p=0.010)
07.sql duration: 6900.90 ms ± 1.75% (std=73.75) delta: 1.09% (p=0.012)
08.sql duration: 6381.42 ms ± 2.24% (std=67.42) delta: 3.22% (p=0.003)
09.sql duration: 22325.80 ms ± 4.73% (std=596.99) delta: -7.49% (p=0.001)
10.sql duration: 4580.01 ms ± 1.79% (std=49.43) delta: 1.63% (p=0.278)
11.sql duration: 1949.66 ms ± 5.48% (std=62.38) delta: 1.98% (p=0.055)
12.sql duration: 6549.94 ms ± 1.59% (std=63.64) delta: 2.35% (p=0.382)
13.sql duration: 6481.35 ms ± 1.08% (std=32.51) delta: -1.49% (p=0.001)
14.sql duration: 5732.91 ms ± 1.76% (std=50.45) delta: 2.85% (p=0.000)
15.sql duration: 10158.18 ms ± 3.15% (std=147.74) delta: 3.01% (p=0.303)
16.sql duration: 2381.46 ms ± 4.81% (std=75.76) delta: 0.14% (p=0.687)
17.sql duration: 17993.45 ms ± 3.39% (std=320.97) delta: 0.87% (p=0.369)
18.sql duration: 26298.73 ms ± 1.55% (std=211.92) delta: 0.42% (p=0.267)
19.sql duration: 7050.49 ms ± 3.07% (std=117.08) delta: 1.00% (p=0.071)
20.sql duration: 5632.63 ms ± 3.69% (std=123.73) delta: -0.44% (p=0.238)
21.sql duration: 11599.09 ms ± 1.21% (std=67.78) delta: 2.36% (p=0.000)
22.sql duration: 3866.24 ms ± 4.67% (std=88.51) delta: -5.55% (p=0.000)

Renkai · 2020-07-20T09:21:46Z

No need to continue because of #8215

ti-srebot added the contribution Type: PR - From contributors label Jul 1, 2020

Renkai added 2 commits July 1, 2020 17:31

use vec executor instead of normal one

606aa9a

Signed-off-by: Renkai <gaelookair@gmail.com>

use vec executor instead of normal one

9b71916

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai force-pushed the batch_handle_index2 branch from ab7e20c to 9b71916 Compare July 1, 2020 09:33

Renkai mentioned this pull request Jul 1, 2020

[WIP] use tidb_query_vec_executors instead of normal executors. #8132

Closed

Renkai added 5 commits July 1, 2020 18:05

bug fix

a22b362

Signed-off-by: Renkai <gaelookair@gmail.com>

Merge branch 'master' into batch_handle_index2

b4bd2ec

Merge branch 'master' into batch_handle_index2

fdd3665

use growing batch size

bb1c4c3

Signed-off-by: Renkai <gaelookair@gmail.com>

use growing batch size

97b8b80

Signed-off-by: Renkai <gaelookair@gmail.com>

breezewish reviewed Jul 3, 2020

View reviewed changes

Renkai added 3 commits July 4, 2020 09:56

Merge branch 'master' into batch_handle_index2

e67b119

Merge branch 'master' into batch_handle_index2

cbf111b

Merge branch 'master' into batch_handle_index2

1469355

Renkai requested a review from breezewish July 7, 2020 03:39

try to use constant batch size

4d7ec0d

Signed-off-by: Renkai <gaelookair@gmail.com>

Merge branch 'master' into batch_handle_index2

1e96bc2

Renkai closed this Jul 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use tidb_query_vec_executors instead of normal executors #8173

Use tidb_query_vec_executors instead of normal executors #8173

Renkai commented Jul 1, 2020

Renkai commented Jul 2, 2020

breezewish Jul 3, 2020

Renkai Jul 4, 2020

breezewish Jul 3, 2020

Renkai Jul 4, 2020

Renkai commented Jul 7, 2020

Renkai commented Jul 7, 2020

Renkai commented Jul 7, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

breezewish commented Jul 13, 2020

mahjonp commented Jul 13, 2020 •

edited

sre-bot commented Jul 13, 2020

sre-bot commented Jul 13, 2020

codecov bot commented Jul 13, 2020

sre-bot commented Jul 13, 2020

Renkai commented Jul 20, 2020

Use tidb_query_vec_executors instead of normal executors #8173

Use tidb_query_vec_executors instead of normal executors #8173

Conversation

Renkai commented Jul 1, 2020

What problem does this PR solve?

Related changes

Check List

Release note

Renkai commented Jul 2, 2020

breezewish Jul 3, 2020

Choose a reason for hiding this comment

Renkai Jul 4, 2020

Choose a reason for hiding this comment

breezewish Jul 3, 2020

Choose a reason for hiding this comment

Renkai Jul 4, 2020

Choose a reason for hiding this comment

Renkai commented Jul 7, 2020

Renkai commented Jul 7, 2020

Renkai commented Jul 7, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

Benchmark Report

sre-bot commented Jul 8, 2020

Benchmark Report

sre-bot commented Jul 8, 2020

Benchmark Report

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

Renkai commented Jul 8, 2020

Renkai commented Jul 8, 2020

ti-srebot commented Jul 8, 2020

sre-bot commented Jul 8, 2020

Benchmark Report

sre-bot commented Jul 8, 2020

Benchmark Report

breezewish commented Jul 13, 2020

mahjonp commented Jul 13, 2020 • edited

sre-bot commented Jul 13, 2020

Benchmark Report

sre-bot commented Jul 13, 2020

Benchmark Report

codecov bot commented Jul 13, 2020

Codecov Report

sre-bot commented Jul 13, 2020

Benchmark Report

Renkai commented Jul 20, 2020

mahjonp commented Jul 13, 2020 •

edited