diff --git a/TOC.md b/TOC.md index 9a0db10ad6fab..25fe95cf9d3ad 100644 --- a/TOC.md +++ b/TOC.md @@ -10,8 +10,10 @@ + Benchmarks - [How to Test TiDB Using Sysbench](/benchmark/benchmark-tidb-using-sysbench.md) - [How to Run TPC-C Test on TiDB](/benchmark/benchmark-tidb-using-tpcc.md) + + [Sysbench Performance Test - v4.0 vs. v3.0](/benchmark/benchmark-sysbench-v4-vs-v3.md) + + [TPC-H Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpch.md) + + [TPC-C Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpcc.md) - [Sysbench Performance Test - v3.0 vs. v2.1](/benchmark/v3.0-performance-benchmarking-with-sysbench.md) - - [TPC-C Performance Test - v4.0 vs. v3.0](/benchmark/v4.0-performance-benchmarking-with-tpcc.md) - [TPC-C Performance Test - v3.0 vs. v2.1](/benchmark/v3.0-performance-benchmarking-with-tpcc.md) - [Interaction Test on Online Workloads and `ADD INDEX` Operations](/benchmark/online-workloads-and-add-index-operations.md) - [Quick Start with TiDB](/quick-start-with-tidb.md) diff --git a/benchmark/benchmark-sysbench-v4-vs-v3.md b/benchmark/benchmark-sysbench-v4-vs-v3.md new file mode 100644 index 0000000000000..6545927a89e29 --- /dev/null +++ b/benchmark/benchmark-sysbench-v4-vs-v3.md @@ -0,0 +1,174 @@ +--- +title: TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0 +summary: Compare the Sysbench performance of TiDB 4.0 and TiDB 3.0. +category: benchmark +--- + +# TiDB Sysbench Performance Test Report -- v4.0 vs. v3.0 + +## Test purpose + +This test aims to compare the Sysbench performance of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. + +## Test environment (AWS EC2) + +### Hardware configuration + +| Service type | EC2 type | Instance count | +|:----------|:----------|:----------| +| PD | m5.xlarge | 3 | +| TiKV | i3.4xlarge| 3 | +| TiDB | c5.4xlarge| 3 | +| Sysbench | m5.4xlarge| 1 | + +### Software version + +| Service type | Software version | +|:----------|:-----------| +| PD | 3.0 and 4.0 | +| TiDB | 3.0 and 4.0 | +| TiKV | 3.0 and 4.0 | +| Sysbench | 1.0.20 | + +### Parameter configuration + +#### TiDB v3.0 configuration + +{{< copyable "" >}} + +```yaml +log.level: "error" +performance.max-procs: 20 +prepared-plan-cache.enabled: true +tikv-client.max-batch-wait-time: 2000000 +``` + +#### TiKV v3.0 configuration + +{{< copyable "" >}} + +```yaml +storage.scheduler-worker-pool-size: 5 +raftstore.store-pool-size: 3 +raftstore.apply-pool-size: 3 +rocksdb.max-background-jobs: 3 +raftdb.max-background-jobs: 3 +raftdb.allow-concurrent-memtable-write: true +server.grpc-concurrency: 6 +readpool.storage.normal-concurrency: 10 +readpool.coprocessor.normal-concurrency: 5 +``` + +#### TiDB v4.0 configuration + +{{< copyable "" >}} + +```yaml +log.level: "error" +performance.max-procs: 20 +prepared-plan-cache.enabled: true +tikv-client.max-batch-wait-time: 2000000 +``` + +#### TiKV v4.0 configuration + +{{< copyable "" >}} + +```yaml +storage.scheduler-worker-pool-size: 5 +raftstore.store-pool-size: 3 +raftstore.apply-pool-size: 3 +rocksdb.max-background-jobs: 3 +raftdb.max-background-jobs: 3 +raftdb.allow-concurrent-memtable-write: true +server.request-batch-enable-cross-command: false +server.grpc-concurrency: 6 +readpool.unifiy-read-pool: true +readpool.unified.min-thread-count: 5 +readpool.unified.max-thread-count: 20 +readpool.storage.normal-concurrency: 10 +pessimistic-txn.pipelined: true +``` + +#### Global variable configuration + +{{< copyable "sql" >}} + +```sql +set global tidb_hashagg_final_concurrency=1; +set global tidb_hashagg_partial_concurrency=1; +set global tidb_disable_txn_auto_retry=0; +``` + +## Test plan + +1. Deploy TiDB v4.0 and v3.0 using TiUP. +2. Use Sysbench to import 16 tables, each table with 10 million rows of data. +3. Execute the `analyze table` statement on each table. +4. Back up the data used for restore before different concurrency tests, which ensures data consistency for each test. +5. Start the Sysbench client to perform the `point_select`, `read_write`, `update_index`, and `update_non_index` tests. Perform stress tests on TiDB via AWS NLB. In each type of test, the warm-up takes 1 minute and the test takes 5 minutes. +6. After each type of test is completed, stop the cluster, overwrite the cluster with the backup data in step 4, and restart the cluster. + +### Prepare test data + +Execute the following command to prepare the test data: + +{{< copyable "shell-regular" >}} + +```bash +sysbench oltp_common \ + --threads=16 \ + --rand-type=uniform \ + --db-driver=mysql \ + --mysql-db=sbtest \ + --mysql-host=$aws_nlb_host \ + --mysql-port=$aws_nlb_port \ + --mysql-user=root \ + --mysql-password=password \ + prepare --tables=16 --table-size=10000000 +``` + +### Perform the test + +Execute the following command to perform the test. + +{{< copyable "shell-regular" >}} + +```bash +sysbench $testname \ + --threads=$threads \ + --time=300 \ + --report-interval=1 \ + --rand-type=uniform \ + --db-driver=mysql \ + --mysql-db=sbtest \ + --mysql-host=$aws_nlb_host \ + --mysql-port=$aws_nlb_port \ + run --tables=16 --table-size=10000000 +``` + +## Test results + +### Point Select performance + +Compared with v3.0, the Point Select performance of TiDB v4.0 has increased by 14%. + +![Point Select](/media/sysbench-v4vsv3-point-select.png) + +### Update Non-index performance + +Compared with v3.0, the Update Non-index performance of TiDB v4.0 has increased by 15%. + +![Update Non-index](/media/sysbench-v4vsv3-update-non-index.png) + +### Update Index performance + +Compared with v3.0, the Update Index performance of TiDB v4.0 has increased by 17%. + +![Update Index](/media/sysbench-v4vsv3-update-index.png) + +### Read-write performance + +Compared with v3.0, the read-write performance of TiDB v4.0 has increased by 31%. + +![Read Write](/media/sysbench-v4vsv3-read-write.png) diff --git a/benchmark/v4.0-performance-benchmarking-with-tpcc.md b/benchmark/v4.0-performance-benchmarking-with-tpcc.md index afe4019feabe1..b535a14ff8c96 100644 --- a/benchmark/v4.0-performance-benchmarking-with-tpcc.md +++ b/benchmark/v4.0-performance-benchmarking-with-tpcc.md @@ -1,113 +1,133 @@ --- title: TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 -summary: Compares the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. +summary: Compare the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. category: benchmark -aliases: ['/docs/dev/benchmark/tpcc-v2/'] --- # TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 ## Test purpose -This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0. +This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. -## Test version, time, and place +## Test environment (AWS EC2) -TiDB versions: v4.0.0-rc vs. v3.0.8 +### Hardware configuration -Time: April, 2020 +| Service type | EC2 type | Instance count | +|:----------|:----------|:----------| +| PD | m5.xlarge | 3 | +| TiKV | i3.4xlarge| 3 | +| TiDB | c5.4xlarge| 3 | +| Sysbench | m5.4xlarge| 1 | -Place: Beijing +### Software version -## Test environment +| Service type | Software version | +|:----------|:-----------| +| PD | 3.0 and 4.0 | +| TiDB | 3.0 and 4.0 | +| TiKV | 3.0 and 4.0 | +| BenchmarkSQL | None | -AWS machine: +### Parameter configuration -| Type | Machine | Number | vCPUs | -| :- | :- | :- | :- | -| TiKV | i3.4xlarge | 3 | 16 | -| TiDB | c5.4xlarge | 2 | 16 | -| PD | m5.xlarge | 3 | 4 | +#### TiDB v3.0 configuration -This test uses the open-source BenchmarkSQL 5.0 as the TPC-C testing tool and adds the support for the MySQL protocol. You can download the testing program by using the following command: - -{{< copyable "shell-regular" >}} +{{< copyable "" >}} -```shell -git clone -b 5.0-mysql-support-opt https://github.com/pingcap/benchmarksql.git +```yaml +log.level: "error" +performance.max-procs: 20 +prepared-plan-cache.enabled: true +tikv-client.max-batch-wait-time: 2000000 ``` -## Test Plan - -Use BenchmarkSQL to load the data of **5000 warehouses** into the TiDB cluster. By using HAProxy, send concurrent requests to the cluster at an incremental number. A single concurrent test lasts 10 minutes. +#### TiKV v3.0 configuration -### TiDB version information +{{< copyable "" >}} -#### v4.0.0-rc +```yaml +storage.scheduler-worker-pool-size: 5 +raftstore.store-pool-size: 3 +raftstore.apply-pool-size: 3 +rocksdb.max-background-jobs: 3 +raftdb.max-background-jobs: 3 +raftdb.allow-concurrent-memtable-write: true +server.grpc-concurrency: 6 +readpool.storage.normal-concurrency: 10 +readpool.coprocessor.normal-concurrency: 5 +``` -| Component | GitHash | -| :- | :- | -| TiDB | 79db9e30ab8f98ac07c8ae55c66dfecc24b43d56 | -| TiKV | f45d0c963df3ee4b1011caf5eb146cacd1fbbad8 | -| PD | 6f06805f3b0070107fcb4af68b2fc224dee0714d | +#### TiDB v4.0 configuration -#### v3.0.8 +{{< copyable "" >}} -| Component | GitHash | -| :- | :- | -| TiDB | 8f13cf1449bd8903ff465a4f12ed89ecbac858a4 | -| TiKV | 0d3168cfd8224fbc48a07796df83ddac0fbcbf46 | -| PD | 456c42b8b0955b33426b58054e43b771801a74d0 | +```yaml +log.level: "error" +performance.max-procs: 20 +prepared-plan-cache.enabled: true +tikv-client.max-batch-wait-time: 2000000 +``` -### TiDB parameter configuration +#### TiKV v4.0 configuration {{< copyable "" >}} -```toml -[log] -level = "error" -[performance] -max-procs = 20 -[prepared_plan_cache] -enabled = true +```yaml +storage.scheduler-worker-pool-size: 5 +raftstore.store-pool-size: 3 +raftstore.apply-pool-size: 3 +rocksdb.max-background-jobs: 3 +raftdb.max-background-jobs: 3 +raftdb.allow-concurrent-memtable-write: true +server.request-batch-enable-cross-command: false +server.grpc-concurrency: 6 +readpool.unifiy-read-pool: true +readpool.unified.min-thread-count: 5 +readpool.unified.max-thread-count: 20 +readpool.storage.normal-concurrency: 10 +pessimistic-txn.pipelined: true ``` -### TiKV parameter configuration +#### Global variable configuration -{{< copyable "" >}} +{{< copyable "sql" >}} -```toml -[readpool] -unify-read-pool = true -[readpool.unified] -min-thread-count = 5 -max-thread-count = 6 -[readpool.storage] -[readpool.coprocessor] -[storage] -scheduler-worker-pool-size = 3 -[raftstore] -store-pool-size = 3 -[rocksdb] -max-background-jobs = 3 -wal-dir = "" -[raftdb] -max-background-jobs = 3 -allow-concurrent-memtable-write = true -[server] -request-batch-enable-cross-command = false -[pessimistic-txn] -pipelined = true +```sql +set global tidb_hashagg_final_concurrency=1; +set global tidb_hashagg_partial_concurrency=1; +set global tidb_disable_txn_auto_retry=0; ``` -### Cluster topology +## Test plan + +1. Deploy TiDB v4.0 and v3.0 using TiUP. + +2. Use BenchmarkSQL to import the TPC-C 5000 Warehouse data. + + 1. Compile BenchmarkSQL: + + {{< copyable "bash" >}} + + ```bash + git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant + ``` + + 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. + + 3. Execute the `runSQL.sh ./props.mysql sql.mysql/tableCreates.sql` command. + + 4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command. + + 5. Run MySQL client and execute the `analyze table` statement on every table. + +3. Execute the `runBenchmark.sh ./props.mysql` command. -* 1 TiKV instance is deployed on each `i3.4xlarge` machine, 3 in total. -* 1 TiDB instance is deployed on each `c5.4xlarge` machine, 2 in total. -* 1 PD instance is deployed on each `m5.xlarge` machine, 3 in total. +4. Extract the tpmC data of New Order from the result. ## Test result -![tpcc](/media/tpcc-3.0-4.0.png) +According to the test statistics, the TPC-C performance of TiDB v4.0 has **increased by 50%** compared with that of TiDB v3.0. -According to the testing statistics, the performance of TiDB 4.0.0-rc **has increased by 45%** than that of TiDB 3.0.8. +![TPC-C](/media/tpcc-v4vsv3.png) diff --git a/benchmark/v4.0-performance-benchmarking-with-tpch.md b/benchmark/v4.0-performance-benchmarking-with-tpch.md new file mode 100644 index 0000000000000..064a43617f8fe --- /dev/null +++ b/benchmark/v4.0-performance-benchmarking-with-tpch.md @@ -0,0 +1,201 @@ +--- +title: TiDB TPC-H Performance Test Report -- v4.0 vs. v3.0 +summary: Compare the TPC-H performance of TiDB 4.0 and TiDB 3.0. +category: benchmark +--- + +# TiDB TPC-H Performance Test Report -- v4.0 vs. v3.0 + +## Test purpose + +This test aims to compare the TPC-H performance of TiDB 4.0 and TiDB 3.0 in the online analytical processing (OLAP) scenario. + +Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0, which enhances TiDB's Hybrid Transactional and Analytical Processing (HTAP) capabilities, test objects in this report are as follows: + ++ TiDB v3.0 that reads data only from TiKV. ++ TiDB v4.0 that reads data only from TiKV. ++ TiDB v4.0 that reads data from TiKV and TiFlash automatically based on intelligent choice. + +## Test environment (AWS EC2) + +### Hardware configuration + +| Service type | EC2 type | Instance count | +|:----------------|:------------|:----| +| PD | m5.xlarge | 3 | +| TiDB | c5.4xlarge | 2 | +| TiKV & TiFlash | i3.4xlarge | 3 | +| TPC-H | m5.xlarge | 1 | + +### Software version + +| Service type | Software version | +|:----------|:-----------| +| PD | 3.0 and 4.0 | +| TiDB | 3.0 and 4.0 | +| TiKV | 3.0 and 4.0 | +| TiFlash | 4.0 | +| tiup-bench | 0.2 | + +### Parameter configuration + +#### v3.0 + +For v3.0, TiDB, TiKV, and PD use the default parameter configuration. + +##### Global variable configuration + +{{< copyable "sql" >}} + +```sql +set global tidb_distsql_scan_concurrency = 30; +set global tidb_projection_concurrency = 16; +set global tidb_hashagg_partial_concurrency = 16; +set global tidb_hashagg_final_concurrency = 16; +set global tidb_hash_join_concurrency = 16; +set global tidb_index_lookup_concurrency = 16; +set global tidb_index_lookup_join_concurrency = 16; +``` + +#### v4.0 + +For v4.0, TiDB uses the default parameter configuration. + +##### TiKV configuration + +{{< copyable "" >}} + +```yaml +readpool.storage.use-unified-pool: false +readpool.coprocessor.use-unified-pool: true +``` + +##### PD configuration + +{{< copyable "" >}} + +```yaml +replication.enable-placement-rules: true +``` + +##### TiFlash configuration + +{{< copyable "" >}} + +```yaml +logger.level: "info" +learner_config.log-level: "info" +``` + +##### Global variable configuration + +{{< copyable "sql" >}} + +```sql +set global tidb_allow_batch_cop = 1; +set global tidb_opt_distinct_agg_push_down = 1; +set global tidb_distsql_scan_concurrency = 30; +set global tidb_projection_concurrency = 16; +set global tidb_hashagg_partial_concurrency = 16; +set global tidb_hashagg_final_concurrency = 16; +set global tidb_hash_join_concurrency = 16; +set global tidb_index_lookup_concurrency = 16; +set global tidb_index_lookup_join_concurrency = 16; +``` + +## Test plan + +### Hardware prerequisite + +To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 to `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. + +### Test process + +1. Deploy TiDB v4.0 and v3.0 using [TiUP](https://pingcap.com/docs/stable/tiup/tiup-overview/#tiup-overview). + +2. Use the bench tool of TiUP to import the TPC-H data with the scale factor 10. + + * Execute the following command to import data into v3.0: + + {{< copyable "bash" >}} + + ```bash + tiup bench tpch prepare \ + --host ${tidb_v3_host} --port ${tidb_v3_port} --db tpch_10 \ + --sf 10 \ + --analyze --tidb_build_stats_concurrency 8 --tidb_distsql_scan_concurrency 30 + ``` + + * Execute the following command to import data into v4.0: + + {{< copyable "bash" >}} + + ```bash + tiup bench tpch prepare \ + --host ${tidb_v4_host} --port ${tidb_v4_port} --db tpch_10 --password ${password} \ + --sf 10 \ + --tiflash \ + --analyze --tidb_build_stats_concurrency 8 --tidb_distsql_scan_concurrency 30 + ``` + +3. Execute the TPC-H queries. + + 1. Download the TPC-H SQL query file: + + {{< copyable "" >}} + + ```bash + git clone https://github.com/pingcap/tidb-bench.git && cd tpch/queries + ``` + + 2. Execute TPC-H queries and record the executing time of each query. + + * For TiDB v3.0, use the MySQL client to connect to TiDB, execute the queries, and record the execution time of each query. + * For TiDB v4.0, use the MySQL client to connect to TiDB, and choose one of the following operations based on where data is read from: + * If data is read only from TiKV, set `set @@session.tidb_isolation_read_engines = 'tikv,tidb';`, execute the queries, and record the execution time of each query. + * If data is read from TiKV and TiFlash automatically based on cost-based intelligent choice, set `set @@session.tidb_isolation_read_engines = 'tikv,tiflash,tidb';`, execute the query, and record the execution time of each query. + +4. Extract and organize the data of query execution time. + +## Test result + +| Query ID | v3.0 | v4.0 TiKV Only | v4.0 TiKV/TiFlash Automatically | +| :-------- | :----------- | :------------ | :-------------- | +| 1 | 7.78 s | 7.45 s | 2.09 s | +| 2 | 3.15 s | 1.71 s | 1.71 s | +| 3 | 6.61 s | 4.10 s | 4.05 s | +| 4 | 2.98 s | 2.56 s | 1.87 s | +| 5 | 20.35 s | 5.71 s | 8.53 s | +| 6 | 4.75 s | 2.44 s | 0.39 s | +| 7 | 7.97 s | 3.72 s | 3.59 s | +| 8 | 5.89 s | 3.22 s | 8.59 s | +| 9 | 34.08 s | 11.87 s | 15.41 s | +| 10 | 4.83 s | 2.75 s | 3.35 s | +| 11 | 3.98 s | 1.60 s | 1.59 s | +| 12 | 5.63 s | 3.40 s | 1.03 s | +| 13 | 5.41 s | 4.56 s | 4.02 s | +| 14 | 5.19 s | 3.10 s | 0.78 s | +| 15 | 10.25 s | 1.82 s | 1.26 s | +| 16 | 2.46 s | 1.51 s | 1.58 s | +| 17 | 23.76 s | 12.38 s | 8.52 s | +| 18 | 17.14 s | 16.38 s | 16.06 s | +| 19 | 5.70 s | 4.59 s | 3.20 s | +| 20 | 4.98 s | 1.89 s | 1.29 s | +| 21 | 11.12 s | 6.23 s | 6.26 s | +| 22 | 4.49 s | 3.05 s | 2.31 s | + +![TPC-H](/media/tpch-v4vsv3.png) + +In the performance diagram above: + ++ Blue lines represent v3.0; ++ Red lines represent v4.0 (data read only from TiKV); ++ Yellow lines represent v4.0 (data read from TiKV and TiFlash automatically based on intelligent choice). ++ The y-axis represents the execution time of the query. The less the time, the better the performance. + +Result description: + ++ **v4.0 TiKV Only** means that TiDB reads data only from TiKV. The result shows that the TPC-H performance increased after TiDB and TiKV are upgraded to v4.0. ++ **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to read data from the TiFlash replica according to the cost estimation. The result shows that the TPC-H performance increased in the full HTAP form of v4.0. + +From the diagram above, you can see that TPC-H performance increases by about 100% on average over a set of 22 queries. diff --git a/media/sysbench-v4vsv3-point-select.png b/media/sysbench-v4vsv3-point-select.png new file mode 100644 index 0000000000000..b9a4cbba03bc3 Binary files /dev/null and b/media/sysbench-v4vsv3-point-select.png differ diff --git a/media/sysbench-v4vsv3-read-write.png b/media/sysbench-v4vsv3-read-write.png new file mode 100644 index 0000000000000..9b3d48ec22d2d Binary files /dev/null and b/media/sysbench-v4vsv3-read-write.png differ diff --git a/media/sysbench-v4vsv3-update-index.png b/media/sysbench-v4vsv3-update-index.png new file mode 100644 index 0000000000000..d58689b519df1 Binary files /dev/null and b/media/sysbench-v4vsv3-update-index.png differ diff --git a/media/sysbench-v4vsv3-update-non-index.png b/media/sysbench-v4vsv3-update-non-index.png new file mode 100644 index 0000000000000..d158d731a26cc Binary files /dev/null and b/media/sysbench-v4vsv3-update-non-index.png differ diff --git a/media/tpcc-v4vsv3.png b/media/tpcc-v4vsv3.png new file mode 100644 index 0000000000000..d33e844b54530 Binary files /dev/null and b/media/tpcc-v4vsv3.png differ diff --git a/media/tpch-v4vsv3.png b/media/tpch-v4vsv3.png new file mode 100644 index 0000000000000..b9ccee3908a3c Binary files /dev/null and b/media/tpch-v4vsv3.png differ