From eac990c0fe16882a2e843574505c1dfaff1987e1 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 17 Jul 2020 14:46:21 +0800 Subject: [PATCH 1/3] cherry pick #3246 to release-3.1 Signed-off-by: ti-srebot --- TOC.md | 136 ++++++++++++++++++++++++++++++++++++++++++++++ column-pruning.md | 20 +++++++ 2 files changed, 156 insertions(+) create mode 100644 column-pruning.md diff --git a/TOC.md b/TOC.md index 561d1696526fa..6b43cfdb8db7f 100644 --- a/TOC.md +++ b/TOC.md @@ -92,6 +92,7 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale +<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -99,6 +100,141 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) +======= + + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) + + [Use TiDB Ansible](/scale-tidb-using-ansible.md) + + [Use TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/scale-a-tidb-cluster) + + Backup and Restore + + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) + + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) + + Use BR Tool + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + [BR storages](/br/backup-and-restore-storages.md) + + [Configure Time Zone](/configure-time-zone.md) + + [Daily Checklist](/daily-check.md) + + [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) + + [Maintain TiFlash](/tiflash/maintain-tiflash.md) + + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) + + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) ++ Monitor and Alert + + [Monitoring Framework Overview](/tidb-monitoring-framework.md) + + [Monitoring API](/tidb-monitoring-api.md) + + [Deploy Monitoring Services](/deploy-monitoring-services.md) + + [TiDB Cluster Alert Rules](/alert-rules.md) + + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) ++ Troubleshoot + + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) + + [Identify Slow Queries](/identify-slow-queries.md) + + [SQL Diagnostics](/system-tables/system-table-sql-diagnostics.md) + + [Identify Expensive Queries](/identify-expensive-queries.md) + + [Statement Summary Tables](/statement-summary-tables.md) + + [Troubleshoot Hotspot Issues](/troubleshoot-hot-spot-issues.md) + + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) + + [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) + + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) + + [Troubleshoot Write Conflicts in Optimistic Transactions](/troubleshoot-write-conflicts.md) ++ Performance Tuning + + System Tuning + + [Operating System Tuning](/tune-operating-system.md) + + Software Tuning + + Configuration + + [Tune TiDB Memory](/configure-memory-usage.md) + + [Tune TiKV Threads](/tune-tikv-thread-performance.md) + + [Tune TiKV Memory](/tune-tikv-memory-performance.md) + + [TiKV Follower Read](/follower-read.md) + + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) + + [Coprocessor Cache](/coprocessor-cache.md) + + SQL Tuning + + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) + + SQL Optimization + + [SQL Optimization Process](/sql-optimization-concepts.md) + + Logic Optimization + + [Subquery Related Optimizations](/subquery-optimization.md) + + [Column Pruning](/column-pruning.md) + + [Decorrelation of Correlated Subquery](/correlated-subquery-optimization.md) + + [Predicates Push Down](/predicates-push-down.md) + + [TopN and Limit Push Down](/topn-limit-push-down.md) + + [Join Reorder](/join-reorder.md) + + Physical Optimization + + [Index Selection](/choose-index.md) + + [Statistics](/statistics.md) + + [Distinct Optimization](/agg-distinct-optimization.md) + + [Prepare Execution Plan Cache](/sql-prepare-plan-cache.md) + + Control Execution Plan + + [Optimizer Hints](/optimizer-hints.md) + + [SQL Plan Management](/sql-plan-management.md) + + [Access Tables Using `IndexMerge`](/index-merge.md) + + [The Blocklist of Optimization Rules and Expression Pushdown](/blocklist-control-plan.md) ++ Tutorials + + [Multiple Data Centers in One City Deployment](/multi-data-centers-in-one-city-deployment.md) + + [Three Data Centers in Two Cities Deployment](/three-data-centers-in-two-cities-deployment.md) + + Best Practices + + [Use TiDB](/tidb-best-practices.md) + + [Java Application Development](/best-practices/java-app-best-practices.md) + + [Use HAProxy](/best-practices/haproxy-best-practices.md) + + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) + + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) + + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) + + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) + + [Use Placement Rules](/configure-placement-rules.md) + + [Use Load Base Split](/configure-load-base-split.md) + + [Use Store Limit](/configure-store-limit.md) ++ TiDB Ecosystem Tools + + [Overview](/ecosystem-tool-user-guide.md) + + [Use Cases](/ecosystem-tool-user-case.md) + + [Download](/download-ecosystem-tools.md) + + Backup & Restore (BR) + + [BR FAQ](/br/backup-and-restore-faq.md) + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + TiDB Binlog + + [Overview](/tidb-binlog/tidb-binlog-overview.md) + + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) + + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) + + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) + + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) + + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) + + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) + + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) + + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) + + [binlogctl](/tidb-binlog/binlog-control.md) + + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) + + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) + + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) + + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) + + Troubleshoot + + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) + + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) + + [FAQ](/tidb-binlog/tidb-binlog-faq.md) + + TiDB Lightning + + [Overview](/tidb-lightning/tidb-lightning-overview.md) + + [Tutorial](/get-started-with-tidb-lightning.md) + + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) + + [Configure](/tidb-lightning/tidb-lightning-configuration.md) + + Key Features + + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) + + [Table Filter](/table-filter.md) + + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) + + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) + + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) + + [Troubleshoot](/troubleshoot-tidb-lightning.md) + + [FAQ](/tidb-lightning/tidb-lightning-faq.md) + + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) + + [TiCDC](/ticdc/ticdc-overview.md) + + sync-diff-inspector + + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) + + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) + + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) + + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) + + [Loader](/loader-overview.md) + + [Mydumper](/mydumper-overview.md) + + [Syncer](/syncer-overview.md) + + TiSpark + + [Quick Start](/get-started-with-tispark.md) + + [User Guide](/tispark-overview.md) +>>>>>>> 15bdb6a... perf-tuning: add column-prune.md (#3246) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) diff --git a/column-pruning.md b/column-pruning.md new file mode 100644 index 0000000000000..d018ab15084a6 --- /dev/null +++ b/column-pruning.md @@ -0,0 +1,20 @@ +--- +title: Column Pruning +summary: Learn about the usage of column pruning in TiDB. +--- + +# Column Pruning + +The basic idea of column pruning is that for columns not used in the operator, the optimizer does not need to retain them during optimization. Removing these columns reduces the use of I/O resources and facilitates the subsequent optimization. The following is an example of column repetition: + +Suppose there are four columns (a, b, c, and d) in table t. You can execute the following statement: + +{{< copyable "sql" >}} + +```sql +select a from t where b> 5 +``` + +In this query, only column a and column b are used, and column c and column d are redundant. Regarding the query plan of this statement, the `Selection` operator uses column b. Then the `DataSource` operator uses columns a and column b. Columns c and column d can be pruned because the `DataSource` operator does not read them. + +Therefore, when TiDB performs a top-down scanning during the logic optimization phase, redundant columns are pruned to reduce waste of resources. This scanning process is called "Column Pruning", corresponding to the `columnPruner` rule. If you want to disable this rule, refer to [The Blocklist of Optimization Rules and Expression Pushdown](/blocklist-control-plan.md). From bdea9abba9770aa58673b8502c38f9ca2f7e960b Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 20 Jul 2020 16:18:43 +0800 Subject: [PATCH 2/3] resolve conflict --- TOC.md | 136 --------------------------------------------------------- 1 file changed, 136 deletions(-) diff --git a/TOC.md b/TOC.md index 6b43cfdb8db7f..561d1696526fa 100644 --- a/TOC.md +++ b/TOC.md @@ -92,7 +92,6 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale -<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -100,141 +99,6 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) -======= - + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) - + [Use TiDB Ansible](/scale-tidb-using-ansible.md) - + [Use TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/scale-a-tidb-cluster) - + Backup and Restore - + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) - + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) - + Use BR Tool - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + [BR storages](/br/backup-and-restore-storages.md) - + [Configure Time Zone](/configure-time-zone.md) - + [Daily Checklist](/daily-check.md) - + [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) - + [Maintain TiFlash](/tiflash/maintain-tiflash.md) - + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) - + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) -+ Monitor and Alert - + [Monitoring Framework Overview](/tidb-monitoring-framework.md) - + [Monitoring API](/tidb-monitoring-api.md) - + [Deploy Monitoring Services](/deploy-monitoring-services.md) - + [TiDB Cluster Alert Rules](/alert-rules.md) - + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) -+ Troubleshoot - + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - + [Identify Slow Queries](/identify-slow-queries.md) - + [SQL Diagnostics](/system-tables/system-table-sql-diagnostics.md) - + [Identify Expensive Queries](/identify-expensive-queries.md) - + [Statement Summary Tables](/statement-summary-tables.md) - + [Troubleshoot Hotspot Issues](/troubleshoot-hot-spot-issues.md) - + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - + [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) - + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) - + [Troubleshoot Write Conflicts in Optimistic Transactions](/troubleshoot-write-conflicts.md) -+ Performance Tuning - + System Tuning - + [Operating System Tuning](/tune-operating-system.md) - + Software Tuning - + Configuration - + [Tune TiDB Memory](/configure-memory-usage.md) - + [Tune TiKV Threads](/tune-tikv-thread-performance.md) - + [Tune TiKV Memory](/tune-tikv-memory-performance.md) - + [TiKV Follower Read](/follower-read.md) - + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) - + [Coprocessor Cache](/coprocessor-cache.md) - + SQL Tuning - + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) - + SQL Optimization - + [SQL Optimization Process](/sql-optimization-concepts.md) - + Logic Optimization - + [Subquery Related Optimizations](/subquery-optimization.md) - + [Column Pruning](/column-pruning.md) - + [Decorrelation of Correlated Subquery](/correlated-subquery-optimization.md) - + [Predicates Push Down](/predicates-push-down.md) - + [TopN and Limit Push Down](/topn-limit-push-down.md) - + [Join Reorder](/join-reorder.md) - + Physical Optimization - + [Index Selection](/choose-index.md) - + [Statistics](/statistics.md) - + [Distinct Optimization](/agg-distinct-optimization.md) - + [Prepare Execution Plan Cache](/sql-prepare-plan-cache.md) - + Control Execution Plan - + [Optimizer Hints](/optimizer-hints.md) - + [SQL Plan Management](/sql-plan-management.md) - + [Access Tables Using `IndexMerge`](/index-merge.md) - + [The Blocklist of Optimization Rules and Expression Pushdown](/blocklist-control-plan.md) -+ Tutorials - + [Multiple Data Centers in One City Deployment](/multi-data-centers-in-one-city-deployment.md) - + [Three Data Centers in Two Cities Deployment](/three-data-centers-in-two-cities-deployment.md) - + Best Practices - + [Use TiDB](/tidb-best-practices.md) - + [Java Application Development](/best-practices/java-app-best-practices.md) - + [Use HAProxy](/best-practices/haproxy-best-practices.md) - + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) - + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) - + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) - + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) - + [Use Placement Rules](/configure-placement-rules.md) - + [Use Load Base Split](/configure-load-base-split.md) - + [Use Store Limit](/configure-store-limit.md) -+ TiDB Ecosystem Tools - + [Overview](/ecosystem-tool-user-guide.md) - + [Use Cases](/ecosystem-tool-user-case.md) - + [Download](/download-ecosystem-tools.md) - + Backup & Restore (BR) - + [BR FAQ](/br/backup-and-restore-faq.md) - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + TiDB Binlog - + [Overview](/tidb-binlog/tidb-binlog-overview.md) - + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) - + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) - + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) - + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) - + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) - + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) - + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) - + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) - + [binlogctl](/tidb-binlog/binlog-control.md) - + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) - + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) - + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) - + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) - + Troubleshoot - + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) - + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) - + [FAQ](/tidb-binlog/tidb-binlog-faq.md) - + TiDB Lightning - + [Overview](/tidb-lightning/tidb-lightning-overview.md) - + [Tutorial](/get-started-with-tidb-lightning.md) - + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) - + [Configure](/tidb-lightning/tidb-lightning-configuration.md) - + Key Features - + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) - + [Table Filter](/table-filter.md) - + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) - + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) - + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) - + [Troubleshoot](/troubleshoot-tidb-lightning.md) - + [FAQ](/tidb-lightning/tidb-lightning-faq.md) - + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) - + [TiCDC](/ticdc/ticdc-overview.md) - + sync-diff-inspector - + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) - + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) - + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) - + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) - + [Loader](/loader-overview.md) - + [Mydumper](/mydumper-overview.md) - + [Syncer](/syncer-overview.md) - + TiSpark - + [Quick Start](/get-started-with-tispark.md) - + [User Guide](/tispark-overview.md) ->>>>>>> 15bdb6a... perf-tuning: add column-prune.md (#3246) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) From 06942901afa74d28bfdc791f3f72ded66b43e484 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 20 Jul 2020 16:23:02 +0800 Subject: [PATCH 3/3] Update TOC.md --- TOC.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TOC.md b/TOC.md index 561d1696526fa..8ceb65ae0c225 100644 --- a/TOC.md +++ b/TOC.md @@ -301,6 +301,7 @@ - [Statement Summary Table](/statement-summary-tables.md) - [Tune TiKV](/tune-tikv-performance.md) - [Operating System Tuning](/tune-operating-system.md) + - [Column Pruning](/column-pruning.md) + Key Monitoring Metrics - [Overview](/grafana-overview-dashboard.md) - [TiDB](/grafana-tidb-dashboard.md)