diff --git a/TOC-tidb-cloud.md b/TOC-tidb-cloud.md index 7d27c8865422e..6ca9badbfa64e 100644 --- a/TOC-tidb-cloud.md +++ b/TOC-tidb-cloud.md @@ -48,6 +48,7 @@ - [Follower Read](/develop/dev-guide-use-follower-read.md) - [Stale Read](/develop/dev-guide-use-stale-read.md) - [HTAP Queries](/develop/dev-guide-hybrid-oltp-and-olap-queries.md) + - [FastScan](/develop/dev-guide-use-fastscan.md) - Transaction - [Overview](/develop/dev-guide-transaction-overview.md) - [Optimistic and Pessimistic Transactions](/develop/dev-guide-optimistic-and-pessimistic-transaction.md) @@ -150,6 +151,7 @@ - [Aggregation](/explain-aggregation.md) - [Views](/explain-views.md) - [Partitions](/explain-partitions.md) + - [Index Merge](/explain-index-merge.md) - SQL Optimization Process - [Overview](/sql-optimization-concepts.md) - Logic Optimization @@ -166,6 +168,7 @@ - [Overview](/sql-physical-optimization.md) - [Index Selection](/choose-index.md) - [Statistics](/statistics.md) + - [Extended Statistics](/extended-statistics.md) - [Wrong Index Solution](/wrong-index-solution.md) - [Distinct Optimization](/agg-distinct-optimization.md) - [Cost Model](/cost-model.md) @@ -289,6 +292,8 @@ - [`EXECUTE`](/sql-statements/sql-statement-execute.md) - [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) - [`EXPLAIN`](/sql-statements/sql-statement-explain.md) + - [`FLASHBACK CLUSTER TO TIMESTAMP`](/sql-statements/sql-statement-flashback-to-timestamp.md) + - [`FLASHBACK DATABASE`](/sql-statements/sql-statement-flashback-database.md) - [`FLASHBACK TABLE`](/sql-statements/sql-statement-flashback-table.md) - [`FLUSH PRIVILEGES`](/sql-statements/sql-statement-flush-privileges.md) - [`FLUSH STATUS`](/sql-statements/sql-statement-flush-status.md) @@ -297,11 +302,13 @@ - [`GRANT `](/sql-statements/sql-statement-grant-role.md) - [`INSERT`](/sql-statements/sql-statement-insert.md) - [`KILL [TIDB]`](/sql-statements/sql-statement-kill.md) + - [`LOCK STATS`](/sql-statements/sql-statement-lock-stats.md) - [`MODIFY COLUMN`](/sql-statements/sql-statement-modify-column.md) - [`PREPARE`](/sql-statements/sql-statement-prepare.md) - [`RECOVER TABLE`](/sql-statements/sql-statement-recover-table.md) - [`RENAME INDEX`](/sql-statements/sql-statement-rename-index.md) - [`RENAME TABLE`](/sql-statements/sql-statement-rename-table.md) + - [`RENAME USER`](/sql-statements/sql-statement-rename-user.md) - [`REPLACE`](/sql-statements/sql-statement-replace.md) - [`REVOKE `](/sql-statements/sql-statement-revoke-privileges.md) - [`REVOKE `](/sql-statements/sql-statement-revoke-role.md) @@ -320,6 +327,7 @@ - [`SHOW CHARACTER SET`](/sql-statements/sql-statement-show-character-set.md) - [`SHOW COLLATION`](/sql-statements/sql-statement-show-collation.md) - [`SHOW [FULL] COLUMNS FROM`](/sql-statements/sql-statement-show-columns-from.md) + - [`SHOW CREATE DATABASE`](/sql-statements/sql-statement-show-create-database.md) - [`SHOW CREATE SEQUENCE`](/sql-statements/sql-statement-show-create-sequence.md) - [`SHOW CREATE TABLE`](/sql-statements/sql-statement-show-create-table.md) - [`SHOW CREATE USER`](/sql-statements/sql-statement-show-create-user.md) @@ -341,6 +349,7 @@ - [`SHOW SCHEMAS`](/sql-statements/sql-statement-show-schemas.md) - [`SHOW STATS_HEALTHY`](/sql-statements/sql-statement-show-stats-healthy.md) - [`SHOW STATS_HISTOGRAMS`](/sql-statements/sql-statement-show-histograms.md) + - [`SHOW STATS_LOCKED`](/sql-statements/sql-statement-show-stats-locked.md) - [`SHOW STATS_META`](/sql-statements/sql-statement-show-stats-meta.md) - [`SHOW STATUS`](/sql-statements/sql-statement-show-status.md) - [`SHOW TABLE NEXT_ROW_ID`](/sql-statements/sql-statement-show-table-next-rowid.md) @@ -355,6 +364,7 @@ - [`TABLE`](/sql-statements/sql-statement-table.md) - [`TRACE`](/sql-statements/sql-statement-trace.md) - [`TRUNCATE`](/sql-statements/sql-statement-truncate.md) + - [`UNLOCK STATS`](/sql-statements/sql-statement-unlock-stats.md) - [`UPDATE`](/sql-statements/sql-statement-update.md) - [`USE`](/sql-statements/sql-statement-use.md) - [`WITH`](/sql-statements/sql-statement-with.md) @@ -409,6 +419,7 @@ - [Usage Scenarios of Stale Read](/stale-read.md) - [Perform Stale Read Using `As OF TIMESTAMP`](/as-of-timestamp.md) - [Perform Stale Read Using `tidb_read_staleness`](/tidb-read-staleness.md) + - [Perform Stale Read Using `tidb_external_ts`](/tidb-external-ts.md) - [Use the `tidb_snapshot` System Variable](/read-historical-data.md) - System Tables - [`mysql`](/mysql-schema.md) @@ -428,6 +439,8 @@ - [`DEADLOCKS`](/information-schema/information-schema-deadlocks.md) - [`ENGINES`](/information-schema/information-schema-engines.md) - [`KEY_COLUMN_USAGE`](/information-schema/information-schema-key-column-usage.md) + - [`MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md) + - [`MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md) - [`PARTITIONS`](/information-schema/information-schema-partitions.md) - [`PROCESSLIST`](/information-schema/information-schema-processlist.md) - [`REFERENTIAL_CONSTRAINTS`](/information-schema/information-schema-referential-constraints.md) @@ -447,9 +460,12 @@ - [`TIKV_REGION_PEERS`](/information-schema/information-schema-tikv-region-peers.md) - [`TIKV_REGION_STATUS`](/information-schema/information-schema-tikv-region-status.md) - [`TIKV_STORE_STATUS`](/information-schema/information-schema-tikv-store-status.md) + - [`USER_ATTRIBUTES`](/information-schema/information-schema-user-attributes.md) - [`USER_PRIVILEGES`](/information-schema/information-schema-user-privileges.md) + - [`VARIABLES_INFO`](/information-schema/information-schema-variables-info.md) - [`VIEWS`](/information-schema/information-schema-views.md) - [Metadata Lock](/metadata-lock.md) + - [Use UUIDs](/best-practices/uuid.md) - [System Variables](/system-variables.md) - Storage Engines - TiKV diff --git a/TOC.md b/TOC.md index 2819f798db512..070e63dd821cb 100644 --- a/TOC.md +++ b/TOC.md @@ -5,8 +5,7 @@ - About TiDB - [TiDB Introduction](/overview.md) - [TiDB 6.5 Release Notes](/releases/release-6.5.0.md) - - [Basic Features](/basic-features.md) - - [Experimental Features](/experimental-features.md) + - [Features](/basic-features.md) - [MySQL Compatibility](/mysql-compatibility.md) - [TiDB Limitations](/tidb-limitations.md) - [Credits](/credits.md) @@ -429,10 +428,12 @@ - [Create a Data Source](/dm/quick-start-create-source.md) - [Manage Data Sources](/dm/dm-manage-source.md) - [Configure Tasks](/dm/dm-task-configuration-guide.md) - - [Table Routing](/dm/dm-key-features.md) - - [Block and Allow Lists](/dm/dm-key-features.md#block-and-allow-table-lists) - - [Binlog Event Filter](/dm/dm-key-features.md#binlog-event-filter) + - [Shard Merge](/dm/dm-shard-merge.md) + - [Table Routing](/dm/dm-table-routing.md) + - [Block and Allow Lists](/dm/dm-block-allow-table-lists.md) + - [Binlog Event Filter](/dm/dm-binlog-event-filter.md) - [Filter DMLs Using SQL Expressions](/dm/feature-expression-filter.md) + - [Online DDL Tool Support](/dm/dm-online-ddl-tool-support.md) - Manage a Data Migration Task - [Precheck a Task](/dm/dm-precheck.md) - [Create a Task](/dm/dm-create-task.md) @@ -564,6 +565,37 @@ - [Storage](/tidb-storage.md) - [Computing](/tidb-computing.md) - [Scheduling](/tidb-scheduling.md) + - Storage Engine - TiKV + - [TiKV Overview](/tikv-overview.md) + - [RocksDB Overview](/storage-engine/rocksdb-overview.md) + - [Titan Overview](/storage-engine/titan-overview.md) + - [Titan Configuration](/storage-engine/titan-configuration.md) + - Storage Engine - TiFlash + - [Overview](/tiflash/tiflash-overview.md) + - [Create TiFlash Replicas](/tiflash/create-tiflash-replicas.md) + - [Use TiDB to Read TiFlash Replicas](/tiflash/use-tidb-to-read-tiflash.md) + - [Use TiSpark to Read TiFlash Replicas](/tiflash/use-tispark-to-read-tiflash.md) + - [Use MPP Mode](/tiflash/use-tiflash-mpp-mode.md) + - [Supported Push-down Calculations](/tiflash/tiflash-supported-pushdown-calculations.md) + - [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) + - [Data Validation](/tiflash/tiflash-data-validation.md) + - [Compatibility](/tiflash/tiflash-compatibility.md) + - [System Variables](/system-variables.md) + - Configuration File Parameters + - [tidb-server](/tidb-configuration-file.md) + - [tikv-server](/tikv-configuration-file.md) + - [tiflash-server](/tiflash/tiflash-configuration.md) + - [pd-server](/pd-configuration-file.md) + - CLI + - [tikv-ctl](/tikv-control.md) + - [pd-ctl](/pd-control.md) + - [tidb-ctl](/tidb-control.md) + - [pd-recover](/pd-recover.md) + - Command Line Flags + - [tidb-server](/command-line-flags-for-tidb-configuration.md) + - [tikv-server](/command-line-flags-for-tikv-configuration.md) + - [tiflash-server](/tiflash/tiflash-command-line-flags.md) + - [pd-server](/command-line-flags-for-pd-configuration.md) - Key Monitoring Metrics - [Overview](/grafana-overview-dashboard.md) - [Performance Overview](/grafana-performance-overview-dashboard.md) @@ -874,38 +906,6 @@ - [Share Session](/dashboard/dashboard-session-share.md) - [Configure SSO](/dashboard/dashboard-session-sso.md) - [FAQ](/dashboard/dashboard-faq.md) - - CLI - - [tikv-ctl](/tikv-control.md) - - [pd-ctl](/pd-control.md) - - [tidb-ctl](/tidb-control.md) - - [pd-recover](/pd-recover.md) - - Command Line Flags - - [tidb-server](/command-line-flags-for-tidb-configuration.md) - - [tikv-server](/command-line-flags-for-tikv-configuration.md) - - [tiflash-server](/tiflash/tiflash-command-line-flags.md) - - [pd-server](/command-line-flags-for-pd-configuration.md) - - Configuration File Parameters - - [tidb-server](/tidb-configuration-file.md) - - [tikv-server](/tikv-configuration-file.md) - - [tiflash-server](/tiflash/tiflash-configuration.md) - - [pd-server](/pd-configuration-file.md) - - [System Variables](/system-variables.md) - - Storage Engines - - TiKV - - [TiKV Overview](/tikv-overview.md) - - [RocksDB Overview](/storage-engine/rocksdb-overview.md) - - [Titan Overview](/storage-engine/titan-overview.md) - - [Titan Configuration](/storage-engine/titan-configuration.md) - - TiFlash - - [Overview](/tiflash/tiflash-overview.md) - - [Create TiFlash Replicas](/tiflash/create-tiflash-replicas.md) - - [Use TiDB to Read TiFlash Replicas](/tiflash/use-tidb-to-read-tiflash.md) - - [Use TiSpark to Read TiFlash Replicas](/tiflash/use-tispark-to-read-tiflash.md) - - [Use MPP Mode](/tiflash/use-tiflash-mpp-mode.md) - - [Supported Push-down Calculations](/tiflash/tiflash-supported-pushdown-calculations.md) - - [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) - - [Data Validation](/tiflash/tiflash-data-validation.md) - - [Compatibility](/tiflash/tiflash-compatibility.md) - [Telemetry](/telemetry.md) - [Errors Codes](/error-codes.md) - [Table Filter](/table-filter.md) diff --git a/basic-features.md b/basic-features.md index 664d5acb685d4..7beec49b30767 100644 --- a/basic-features.md +++ b/basic-features.md @@ -1,12 +1,18 @@ --- title: TiDB Features -summary: Learn about the basic features of TiDB. -aliases: ['/docs/dev/basic-features/'] +summary: Learn about the feature overview of TiDB. +aliases: ['/docs/dev/basic-features/','/tidb/dev/experimental-features-4.0/'] --- # TiDB Features -This document lists the features supported in each TiDB version. Note that supports for experimental features might change before the final release. +This document lists the features supported in each TiDB version. + +> **Note:** +> +> - Y: the feature is generally available (GA) and can be used in production environments. +> - N: the feature is not supported. +> - Experimental: the feature is not GA yet and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice. The syntax and implementation might be modified before the general availability. If you encounter any problems, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. ## Data types, functions, and operators @@ -38,8 +44,11 @@ This document lists the features supported in each TiDB version. Note that suppo | ------------------------------------------------------------ | :--: | :--: | :--: | :--: | :--: | :--: | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | | [Expression indexes](/sql-statements/sql-statement-create-index.md#expression-index) [^2] | Y | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | | [Columnar storage (TiFlash)](/tiflash/tiflash-overview.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | +| [Use FastScan to accelerate queries in OLAP scenarios](/develop/dev-guide-use-fastscan.md) | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | N | N | N | | [RocksDB engine](/storage-engine/rocksdb-overview.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Titan plugin](/storage-engine/titan-overview.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | +| [Titan Level Merge](/storage-engine/titan-configuration.md#level-merge-experimental) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Use buckets to improve scan concurrency](/tune-region-performance.md#use-bucket-to-increase-concurrency) | Experimental | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | N | N | | [Invisible indexes](/sql-statements/sql-statement-add-index.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | | [Composite `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Unique indexes](/constraints.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | @@ -67,6 +76,8 @@ This document lists the features supported in each TiDB version. Note that suppo | [`BATCH [ON COLUMN] LIMIT INTEGER DELETE`](/sql-statements/sql-statement-batch.md) | Y | Y | Y | Y | Y | N | N | N | N | N | N | N | | [`BATCH [ON COLUMN] LIMIT INTEGER INSERT/UPDATE/REPLACE`](/sql-statements/sql-statement-batch.md) | Y | N | N | N | N | N | N | N | N | N | N | N | | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Experimental | N | N | N | N | N | N | N | +| [Table Lock](/tidb-configuration-file.md#enable-table-lock-new-in-v400) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Experimental | N | N | N | N | N | N | N | N | N | N | N | ## Advanced SQL features @@ -74,6 +85,7 @@ This document lists the features supported in each TiDB version. Note that suppo | ------------------------------------------------------------ | :--: | :--: | :--: | :--: | :--: | :--: | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | | [Prepared statement cache](/sql-prepared-plan-cache.md) | Y | Y | Y | Y | Y | Y | Y | Y | Experimental | Experimental | Experimental | Experimental | | [SQL plan management (SPM)](/sql-plan-management.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | +| [Create bindings according to historical execution plans](/sql-plan-management.md#create-a-binding-according-to-a-historical-execution-plan) | Experimental | N | N | N | N | N | N | N | N | N | N | N | | [Coprocessor cache](/coprocessor-cache.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Experimental | | [Stale Read](/stale-read.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | N | | [Follower reads](/follower-read.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | @@ -82,6 +94,7 @@ This document lists the features supported in each TiDB version. Note that suppo | [MPP Execution Engine](/explain-mpp.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | N | | [Index Merge](/explain-index-merge.md) | Y | Y | Y | Y | Y | Y | Y | Experimental | Experimental | Experimental | Experimental | Experimental | | [Placement Rules in SQL](/placement-rules-in-sql.md) | Y | Y | Y | Y | Y | Y | Experimental | Experimental | N | N | N | N | +| [Cascades Planner](/system-variables.md#tidb_enable_cascades_planner) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | ## Data definition language (DDL) @@ -139,6 +152,10 @@ This document lists the features supported in each TiDB version. Note that suppo | [Automatically update statistics](/statistics.md#automatic-update) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Fast Analyze](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental| Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | | [Dynamic pruning](/partitioned-table.md#dynamic-pruning-mode) | Y | Y | Y | Y | Y | Experimental | Experimental | Experimental | Experimental | Experimental | N | N | +| [Collect statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-on-some-columns) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | +| [Control the memory quota for collecting statistics](/statistics.md#the-memory-quota-for-collecting-statistics) | Experimental | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | N | N | +| [Randomly sample about 10000 rows of data to quickly build statistics](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Lock statistics](/statistics.md#lock-statistics) | Experimental | N | N | N | N | N | N | N | N | N | N | N | ## Security @@ -180,6 +197,7 @@ This document lists the features supported in each TiDB version. Note that suppo | [TiDB Dashboard Top SQL](/dashboard/top-sql.md) | Y | Y | Y | Y | Y | Y | Experimental | N | N | N | N | N | | [TiDB Dashboard SQL Diagnostics](/information-schema/information-schema-sql-diagnostics.md) | Y | Y | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | | [TiDB Dashboard Cluster Diagnostics](/dashboard/dashboard-diagnostics-access.md) | Y | Y | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [TiKV-FastTune dashboard](/grafana-tikv-dashboard.md#tikv-fasttune-dashboard) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | | [Information schema](/information-schema/information-schema.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Metrics schema](/metrics-schema.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Statements summary tables](/statement-summary-tables.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | @@ -193,9 +211,12 @@ This document lists the features supported in each TiDB version. Note that suppo | [`SET CONFIG`](/dynamic-config.md) | Y | Y | Y | Y | Y | Experimental| Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | | [DM WebUI](/dm/dm-webui-guide.md) | Experimental| Experimental| Experimental | Experimental| Experimental | Experimental | N | N | N | N | N | N | | [Foreground Quota Limiter](/tikv-configuration-file.md#foreground-quota-limiter) | Y | Y | Y | Y| Experimental | Experimental | N | N | N | N | N | N | +| [Background Quota Limiter](/tikv-configuration-file.md#background-quota-limiter) | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | N | N | N | | [EBS volume snapshot backup and restore](https://docs.pingcap.com/tidb-in-kubernetes/v1.4/backup-to-aws-s3-by-snapshot) | Y | Y | N | N | N | N | N | N | N | N | N | N | | [PITR](/br/backup-and-restore-overview.md) | Y | Y | Y | Y | N | N | N | N | N | N | N | N | | [Global memory control](/configure-memory-usage.md#configure-the-memory-usage-threshold-of-a-tidb-server-instance) | Y | Experimental | N | N | N | N | N | N | N | N | N | N | +| [Cross-cluster RawKV replication](/tikv-configuration-file.md#api-version-new-in-v610) | Experimental | Experimental | Experimental | Experimental | N | N | N | N | N | N | N | N | +| [Green GC](/system-variables.md#tidb_gc_scan_lock_mode-new-in-v50) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | N | [^1]: TiDB incorrectly treats latin1 as a subset of utf8. See [TiDB #18955](https://github.com/pingcap/tidb/issues/18955) for more details. diff --git a/best-practices/uuid.md b/best-practices/uuid.md index 1f2f6d4c2c302..34c6308db8222 100644 --- a/best-practices/uuid.md +++ b/best-practices/uuid.md @@ -21,14 +21,36 @@ The textual UUID format looks like this: `ab06f63e-8fe7-11ec-a514-5405db7aad56`, ### UUID format binary order and a clustered PK -The `UUID_TO_BIN()` function can be used with one argument, the UUID or with two arguments where the second argument is a `swap_flag`. It is recommended to not set the `swap_flag` with TiDB to avoid [hotspots](/best-practices/high-concurrency-best-practices.md). +The `UUID_TO_BIN()` function can be used with one argument, the UUID or with two arguments where the second argument is a `swap_flag`. + + + +It is recommended to not set the `swap_flag` with TiDB to avoid [hotspots](/best-practices/high-concurrency-best-practices.md). + + + + + +It is recommended to not set the `swap_flag` with TiDB to avoid hotspots. + + You can also explicitly set the [`CLUSTERED` option](/clustered-indexes.md) for UUID based primary keys to avoid hotspots. To demonstrate the effect of the `swap_flag`, here are two tables with an identical structure. The difference is that the data inserted into `uuid_demo_1` uses `UUID_TO_BIN(?, 0)` and `uuid_demo_2` uses `UUID_TO_BIN(?, 1)`. + + In the screenshot of the [Key Visualizer](/dashboard/dashboard-key-visualizer.md) below, you can see that writes are concentrated in a single region of the `uuid_demo_2` table that has the order of the fields swapped in the binary format. + + + + +In the screenshot of the [Key Visualizer](/tidb-cloud/tune-performance.md#key-visualizer) below, you can see that writes are concentrated in a single region of the `uuid_demo_2` table that has the order of the fields swapped in the binary format. + + + ![Key Visualizer](/media/best-practices/uuid_keyviz.png) ```sql diff --git a/dashboard/dashboard-cluster-info.md b/dashboard/dashboard-cluster-info.md index 1f1b20c2d685a..e21640f1dd3cf 100644 --- a/dashboard/dashboard-cluster-info.md +++ b/dashboard/dashboard-cluster-info.md @@ -14,7 +14,7 @@ You can use one of the following two methods to access the cluster information p - After logging into TiDB Dashboard, click **Cluster Info** on the left navigation menu: - ![Access cluster information page](/media/dashboard/dashboard-cluster-info-access.png) + ![Access cluster information page](/media/dashboard/dashboard-cluster-info-access-v650.png) - Visit in your browser. Replace `127.0.0.1:2379` with the actual PD instance address and port. @@ -22,7 +22,7 @@ You can use one of the following two methods to access the cluster information p Click **Instances** to view the list of instances: -![Instance list](/media/dashboard/dashboard-cluster-info-instances.png) +![Instance list](/media/dashboard/dashboard-cluster-info-instances-v650.png) This instance list shows the overview information of all instances of TiDB, TiKV, PD, and TiFlash components in the cluster. @@ -32,8 +32,8 @@ The list includes the following information: - Status: The running status of the instance. - Up Time: The start time of the instance. - Version: The instance version number. -- Deployment directory: The directory in which the instance binary file is located. - Git Hash: The Git Hash value corresponding to the instance binary file. +- Deployment directory: The directory in which the instance binary file is located. ### Instance status @@ -56,20 +56,35 @@ Instance status derives from the PD scheduling information. For more details, se Click **Hosts** to view the list of hosts: -![Host list](/media/dashboard/dashboard-cluster-info-hosts.png) +![Host list](/media/dashboard/dashboard-cluster-info-hosts-v650.png) This host list shows the running status of hosts that correspond to all instances of TiDB, TiKV, PD, and TiFlash components in the cluster. The list includes the following information: -- Address: The Host IP address. +- Host Address: The Host IP address. - CPU: The number of logical cores of the host CPU. - CPU Usage: The user-mode and kernel-mode CPU usage in the current 1 second. - Memory: The total physical memory size of the host. - Memory Usage: The current memory usage of the host. -- Disk: The file system of the disk on the host on which the instance is running and the mounting path of this disk. -- Disk Usage: The space usage of the disk on the host on which the instance is running. > **Note:** > > The host list information is provided by each instance process, so when all instances on the host are down, the host information is not displayed. + +## Disk list + +Click **Disks** to view the list of disks: + +![Disk list](/media/dashboard/dashboard-cluster-info-disks-v650.png) + +This disk list shows the status of disks on which the TiDB, TiKV, PD, and TiFlash instances run. + +The list includes the following information: + +- Host Address: The Host IP address. +- Mount Directory: The mount path of this disk on the host on which the instance is running. +- File System: The file system type of this disk on the host on which the instance is running. +- Disk Capacity: The total space of the disk on the host on which the instance is running. +- Disk Usage: The space usage of the disk on the host on which the instance is running. +- Instance: The instance running on this host. diff --git a/dashboard/dashboard-diagnostics-access.md b/dashboard/dashboard-diagnostics-access.md index 9c7402b6eece4..09d33e91b5c45 100644 --- a/dashboard/dashboard-diagnostics-access.md +++ b/dashboard/dashboard-diagnostics-access.md @@ -18,7 +18,7 @@ You can use one of the following methods to access the cluster diagnostics page: - After logging into TiDB Dashboard, click **Cluster Diagnostics** on the left navigation menu: - ![Access Cluster Diagnostics page](/media/dashboard/dashboard-diagnostics-access.png) + ![Access Cluster Diagnostics page](/media/dashboard/dashboard-diagnostics-access-v650.png) - Visit `http://127.0.0.1:2379/dashboard/#/diagnose` in your browser. Replace `127.0.0.1:2379` with the actual PD address and port number. @@ -26,36 +26,36 @@ You can use one of the following methods to access the cluster diagnostics page: To diagnose a cluster within a specified time range and check the cluster load, you can take the following steps to generate a diagnostic report: -1. Set the **Range Start Time**, such as `2020-05-21 14:40:00`. +1. Set the **Range Start Time**, such as `2022-05-21 14:40:00`. 2. Set the **Range Duration**, such as `10 min`. 3. Click **Start**. -![Generate diagnostic report](/media/dashboard/dashboard-diagnostics-gen-report.png) +![Generate diagnostic report](/media/dashboard/dashboard-diagnostics-gen-report-v650.png) > **Note:** > > It is recommended that the **Range Duration** of the report is between 1 minute and 60 minutes. This **Range Duration** cannot exceed 60 minutes. -The steps above generate a diagnostic report for the time range from `2020-05-21 14:40:00` to `2020-05-21 14:50:00`. After clicking **Start**, you can see the interface below. **Progress** is the progress bar of the diagnostic report. After the report is generated, click **View Full Report**. +The preceding steps generate a diagnostic report for the time range from `2022-05-21 14:40:00` to `2022-05-21 14:50:00`. After clicking **Start**, you can see the interface below. **Progress** is the progress bar of the diagnostic report. After the report is generated, click **View Full Report**. -![Report progress](/media/dashboard/dashboard-diagnostics-gen-process.png) +![Report progress](/media/dashboard/dashboard-diagnostics-gen-process-v650.png) ## Generate comparison report If a system exception occurs at a certain point, for example, QPS jitter or higher latency, a diagnostic report can be generated. Particularly, this report compares the system in the abnormal time range with the system in the normal time range. For example: -- Abnormal time range: `2020-05-21 14:40:00`-`2020-05-21 14:45:00`. Within this time range, the system is abnormal. -- Normal time range: `2020-05-21 14:30:00` - `2020-05-21 14:35:00`. Within this time range, the system is normal. +- Abnormal time range: `2022-05-21 14:40:00`-`2022-05-21 14:45:00`. Within this time range, the system is abnormal. +- Normal time range: `2022-05-21 14:30:00` - `2022-05-21 14:35:00`. Within this time range, the system is normal. -You can take the following steps to generate a comparison report for the two time ranges above: +You can take the following steps to generate a comparison report for the preceding two time ranges: -1. Set the **Range Start Time**, which is the start time of the range in which the system becomes abnormal, such as `2020-05-21 14:40:00`. +1. Set the **Range Start Time**, which is the start time of the range in which the system becomes abnormal, such as `2022-05-21 14:40:00`. 2. Set the **Range Duration**. Generally, this duration is the duration of system anomalies, such as 5 minutes. -3. Enable **Compare by baseline**. -4. Set the **Baseline Range Start Time**, which is the start time of the range (to be compared with) in which the system is normal, such as `2020-05-21 14:30:00`. +3. Enable **Compare by Baseline**. +4. Set the **Baseline Range Start Time**, which is the start time of the range (to be compared with) in which the system is normal, such as `2022-05-21 14:30:00`. 5. Click **Start**. -![Generate comparison report](/media/dashboard/dashboard-diagnostics-gen-compare-report.png) +![Generate comparison report](/media/dashboard/dashboard-diagnostics-gen-compare-report-v650.png) Then wait for the report to be generated and click **View Full Report**. diff --git a/dashboard/dashboard-key-visualizer.md b/dashboard/dashboard-key-visualizer.md index da1350097af33..b774daa903096 100644 --- a/dashboard/dashboard-key-visualizer.md +++ b/dashboard/dashboard-key-visualizer.md @@ -14,7 +14,7 @@ You can use one of the following two methods to access the Key Visualizer page: * After logging into TiDB Dashboard, click **Key Visualizer** on the left navigation menu: -![Access Key Visualizer](/media/dashboard/dashboard-keyviz-access.png) + ![Access Key Visualizer](/media/dashboard/dashboard-keyviz-access-v650.png) * Visit in your browser. Replace `127.0.0.1:2379` with the actual PD instance address and port. @@ -24,11 +24,11 @@ The following image is a demonstration of the Key Visualizer page: ![Key Visualizer page](/media/dashboard/dashboard-keyviz-overview.png) -From the interface above, you can see the following objects: +From the preceding interface, you can see the following objects: + A large heatmap that shows changes of the overall traffic over time. + The detailed information of a certain coordinate point. -+ Information of tables, indexes, and so on (on the left side of the heatmap). ++ Information of tables, and indexes (on the left side of the heatmap). ## Basic concepts @@ -94,7 +94,7 @@ When you open Key Visualizer, the heatmap of the entire database over the recent ![Selection box](/media/dashboard/dashboard-keyviz-select-zoom.gif) + Click the **Reset** button to reset the Region range to the entire database. - + Click the **time selection box** (at the position of `6 hours` on the interface above) and select the observation time period again. + + Click the **time selection box** (at the position of `6 hour` on the preceding interface) and select the observation time period again. ![Select time](/media/dashboard/dashboard-keyviz-select-time.png) diff --git a/dashboard/dashboard-metrics-relation.md b/dashboard/dashboard-metrics-relation.md index f278e0e372e27..1c017a0f5aa93 100644 --- a/dashboard/dashboard-metrics-relation.md +++ b/dashboard/dashboard-metrics-relation.md @@ -11,9 +11,9 @@ TiDB Dashboard metrics relation graph is a feature introduced in v4.0.7. This fe Log into TiDB Dashboard, click **Cluster Diagnostics** on the left navigation menu, and you can see the page of generating the metrics relation graph. -![Metrics relation graph homepage](/media/dashboard/dashboard-metrics-relation-home.png) +![Metrics relation graph homepage](/media/dashboard/dashboard-metrics-relation-home-v650.png) -After setting **Range Start Time** and **Range Duration**, click the **Generate Metrics Relation** button and you will enter the page of metrics relation graph. +After setting **Range Start Time** and **Range Duration**, click **Generate Metrics Relation** and you will enter the page of metrics relation graph. ## Understand graph diff --git a/dashboard/dashboard-overview.md b/dashboard/dashboard-overview.md index 37c772d46a8a9..4f993d995b809 100644 --- a/dashboard/dashboard-overview.md +++ b/dashboard/dashboard-overview.md @@ -19,7 +19,7 @@ This page shows the overview of the entire TiDB cluster, including the following After logging into TiDB Dashboard, the overview page is entered by default, or you can click **Overview** on the left navigation menu to enter this page: -![Enter overview page](/media/dashboard/dashboard-overview-access.png) +![Enter overview page](/media/dashboard/dashboard-overview-access-v650.png) ## QPS @@ -73,7 +73,7 @@ This area summarizes the total number of instances and abnormal instances of TiD ![Instances](/media/dashboard/dashboard-overview-instances.png) -The statuses in the image above are described as follows: +The statuses in the preceding image are described as follows: - Up: The instance is running properly (including the offline storage instance). - Down: The instance is running abnormally, such as network disconnection and process crash. diff --git a/dashboard/dashboard-session-share.md b/dashboard/dashboard-session-share.md index 53a7fee124810..dfd4ddb646980 100644 --- a/dashboard/dashboard-session-share.md +++ b/dashboard/dashboard-session-share.md @@ -7,7 +7,7 @@ summary: Learn how to share the current TiDB Dashboard session to other users. You can share the current session of the TiDB Dashboard to other users so that they can access and operate the TiDB Dashboard without entering the user password. -## Steps for the Inviter +## Steps for the inviter 1. Sign into TiDB Dashboard. @@ -15,7 +15,7 @@ You can share the current session of the TiDB Dashboard to other users so that t 3. Click **Share Current Session**. - ![Sample Step](/media/dashboard/dashboard-session-share-settings-1.png) + ![Sample Step](/media/dashboard/dashboard-session-share-settings-1-v650.png) > **Note:** > @@ -29,28 +29,28 @@ You can share the current session of the TiDB Dashboard to other users so that t 5. Click **Generate Authorization Code**. - ![Sample Step](/media/dashboard/dashboard-session-share-settings-2.png) + ![Sample Step](/media/dashboard/dashboard-session-share-settings-2-v650.png) 6. Provide the generated **Authorization Code** to the user to whom you want to share the session. - ![Sample Step](/media/dashboard/dashboard-session-share-settings-3.png) + ![Sample Step](/media/dashboard/dashboard-session-share-settings-3-v650.png) > **Warning:** > > Keep your authorization code secure and do not send it to anyone who is untrusted. Otherwise, they will be able to access and operate TiDB Dashboard without your authorization. -## Steps for the Invitee +## Steps for the invitee 1. On the sign-in page of TiDB Dashboard, click **Use Alternative Authentication**. - ![Sample Step](/media/dashboard/dashboard-session-share-signin-1.png) + ![Sample Step](/media/dashboard/dashboard-session-share-signin-1-v650.png) 2. Click **Authorization Code** to use it to sign in. - ![Sample Step](/media/dashboard/dashboard-session-share-signin-2.png) + ![Sample Step](/media/dashboard/dashboard-session-share-signin-2-v650.png) 3. Enter the authorization code you have received from the inviter. 4. Click **Sign In**. - ![Sample Step](/media/dashboard/dashboard-session-share-signin-3.png) + ![Sample Step](/media/dashboard/dashboard-session-share-signin-3-v650.png) diff --git a/develop/dev-guide-choose-driver-or-orm.md b/develop/dev-guide-choose-driver-or-orm.md index f9ca3c8ec4210..21651c4695578 100644 --- a/develop/dev-guide-choose-driver-or-orm.md +++ b/develop/dev-guide-choose-driver-or-orm.md @@ -14,21 +14,7 @@ summary: Learn how to choose a driver or ORM framework to connect to TiDB. > > For more information, refer to [Third-Party Tools Supported by TiDB](/develop/dev-guide-third-party-support.md). -TiDB is highly compatible with the MySQL protocol but some features are incompatible with MySQL. - -For example: - -- Features that are not supported by TiDB: - - - Stored procedures and functions - - Triggers - - `FOREIGN KEY` constraints - -- Features that are different from MySQL: - - - Auto-increment ID: auto-incremental columns are globally unique in TiDB. They are incremental on a single TiDB server, but ***not*** necessarily incremental among multiple TiDB servers or allocated sequentially. - -For a full list of compatibility differences, see [MySQL Compatibility](/mysql-compatibility.md). +TiDB is highly compatible with the MySQL protocol but some features are incompatible with MySQL. For a full list of compatibility differences, see [MySQL Compatibility](/mysql-compatibility.md). ## Java @@ -305,4 +291,10 @@ Support level: **Compatible** For an example of using peewee to build a TiDB application, see [Build a Simple CRUD App with TiDB and Python](/develop/dev-guide-sample-application-python.md#step-2-get-the-code). - \ No newline at end of file + + + + +After you have determined the driver or ORM, you can [connect to your TiDB cluster](https://docs.pingcap.com/tidbcloud/connect-to-tidb-cluster). + + diff --git a/dm/dm-binlog-event-filter.md b/dm/dm-binlog-event-filter.md new file mode 100644 index 0000000000000..8ce2a0d9c1e9c --- /dev/null +++ b/dm/dm-binlog-event-filter.md @@ -0,0 +1,150 @@ +--- +title: TiDB Data Migration Binlog Event Filter +summary: Learn how to use the binlog event filter feature of DM. +--- + +# TiDB Data Migration Binlog Event Filter + +TiDB Data Migration (DM) provides the binlog event filter feature to filter out or only receive specified types of binlog events for some schemas or tables. For example, you can filter out all `TRUNCATE TABLE` or `INSERT` events. The binlog event filter feature is more fine-grained than the [block and allow lists](/dm/dm-block-allow-table-lists.md) feature. + +## Configure the binlog event filter + +In the task configuration file, add the following configuration: + +```yaml +filters: + rule-1: + schema-pattern: "test_*" + ​table-pattern: "t_*" + ​events: ["truncate table", "drop table"] + sql-pattern: ["^DROP\\s+PROCEDURE", "^CREATE\\s+PROCEDURE"] + ​action: Ignore +``` + +Starting from DM v2.0.2, you can configure the binlog event filter in the source configuration file. For details, see [Upstream Database Configuration File](/dm/dm-source-configuration-file.md). + +In simple scenarios, it is recommended that you use the wildcard for matching schemas and tables. However, note the following version differences: + +- For DM v1.0.5 or later versions, the binlog event filter supports the [wildcard match](https://en.wikipedia.org/wiki/Glob_(programming)#Syntax), but there can be **only one** `*` in the wildcard expression, and `*` **must be placed at the end**. + +- For DM versions earlier than v1.0.5, the binlog event filter supports the wildcard but does not support the `[...]` and `[!...]` expressions. + +## Parameter descriptions + +- [`schema-pattern`/`table-pattern`](/dm/table-selector.md): the binlog events or DDL SQL statements of upstream MySQL or MariaDB instance tables that match `schema-pattern`/`table-pattern` are filtered by the rules below. + +- `events`: the binlog event array. You can only select one or more `Event`s from the following table: + + | Events | Type | Description | + | --------------- | ---- | ----------------------------- | + | `all` | | Includes all the events below | + | `all dml` | | Includes all DML events below | + | `all ddl` | | Includes all DDL events below | + | `none` | | Includes none of the events below | + | `none ddl` | | Includes none of the DDL events below | + | `none dml` | | Includes none of the DML events below | + | `insert` | DML | The `INSERT` DML event | + | `update` | DML | The `UPDATE` DML event | + | `delete` | DML | The `DELETE` DML event | + | `create database` | DDL | The `CREATE DATABASE` DDL event | + | `drop database` | DDL | The `DROP DATABASE` DDL event | + | `create table` | DDL | The `CREATE TABLE` DDL event | + | `create index` | DDL | The `CREATE INDEX` DDL event | + | `drop table` | DDL | The `DROP TABLE` DDL event | + | `truncate table` | DDL | The `TRUNCATE TABLE` DDL event | + | `rename table` | DDL | The `RENAME TABLE` DDL event | + | `drop index` | DDL | The `DROP INDEX` DDL event | + | `alter table` | DDL | The `ALTER TABLE` DDL event | + +- `sql-pattern`: it is used to filter specified DDL SQL statements. The matching rule supports using a regular expression. For example, `"^DROP\\s+PROCEDURE"`. + +- `action`: the string (`Do`/`Ignore`). Based on the following rules, it judges whether to filter. If either of the two rules is satisfied, the binlog is filtered; otherwise, the binlog is not filtered. + + - `Do`: the allow list. The binlog is filtered in either of the following two conditions: + - The type of the event is not in the `event` list of the rule. + - The SQL statement of the event cannot be matched by `sql-pattern` of the rule. + - `Ignore`: the block list. The binlog is filtered in either of the following two conditions: + - The type of the event is in the `event` list of the rule. + - The SQL statement of the event can be matched by `sql-pattern` of the rule. + - When multiple rules match the same table, the rules are applied sequentially. The block list has a higher priority than the allow list. For example, if both the `Ignore` and `Do` rules are applied to the same table, the `Ignore` rule takes effect. + +## Usage examples + +This section shows the usage examples in the scenario of sharding (sharded schemas and tables). + +### Filter all sharding deletion operations + +To filter out all deletion operations, configure the following two filtering rules: + +- `filter-table-rule` filters out the `TRUNCATE TABLE`, `DROP TABLE` and `DELETE STATEMENT` operations of all tables that match the `test_*`.`t_*` pattern. +- `filter-schema-rule` filters out the `DROP DATABASE` operation of all schemas that match the `test_*` pattern. + +```yaml +filters: + filter-table-rule: + schema-pattern: "test_*" + table-pattern: "t_*" + events: ["truncate table", "drop table", "delete"] + action: Ignore + filter-schema-rule: + schema-pattern: "test_*" + events: ["drop database"] + action: Ignore +``` + +### Only migrate sharding DML statements + +To only migrate sharding DML statements, configure the following two filtering rules: + +- `do-table-rule` only migrates the `CREATE TABLE`, `INSERT`, `UPDATE` and `DELETE` statements of all tables that match the `test_*`.`t_*` pattern. +- `do-schema-rule` only migrates the `CREATE DATABASE` statement of all schemas that match the `test_*` pattern. + +> **Note:** +> +> The reason why the `CREATE DATABASE/TABLE` statement is migrated is that you can migrate DML statements only after the schema and table are created. + +```yaml +filters: + do-table-rule: + schema-pattern: "test_*" + table-pattern: "t_*" + events: ["create table", "all dml"] + action: Do + do-schema-rule: + schema-pattern: "test_*" + events: ["create database"] + action: Do +``` + +### Filter out the SQL statements that TiDB does not support + +To filter out the `PROCEDURE` statements that TiDB does not support, configure the following `filter-procedure-rule`: + +```yaml +filters: + filter-procedure-rule: + schema-pattern: "test_*" + table-pattern: "t_*" + sql-pattern: ["^DROP\\s+PROCEDURE", "^CREATE\\s+PROCEDURE"] + action: Ignore +``` + +`filter-procedure-rule` filters out the `^CREATE\\s+PROCEDURE` and `^DROP\\s+PROCEDURE` statements of all tables that match the `test_*`.`t_*` pattern. + +### Filter out the SQL statements that the TiDB parser does not support + +For the SQL statements that the TiDB parser does not support, DM cannot parse them and get the `schema`/`table` information. So you must use the global filtering rule: `schema-pattern: "*"`. + +> **Note:** +> +> To avoid filtering out data that need to be migrated, you must configure the global filtering rule as strictly as possible. + +To filter out the `PARTITION` statements that the TiDB parser (of some version) does not support, configure the following filtering rule: + +```yaml +filters: + filter-partition-rule: + schema-pattern: "*" + sql-pattern: ["ALTER\\s+TABLE[\\s\\S]*ADD\\s+PARTITION", "ALTER\\s+TABLE[\\s\\S]*DROP\\s+PARTITION"] + action: Ignore +``` diff --git a/dm/dm-block-allow-table-lists.md b/dm/dm-block-allow-table-lists.md new file mode 100644 index 0000000000000..0f09f40da5a21 --- /dev/null +++ b/dm/dm-block-allow-table-lists.md @@ -0,0 +1,142 @@ +--- +title: TiDB Data Migration Block and Allow Lists +summary: Learn how to use the DM block and allow lists feature. +--- + +# TiDB Data Migration Block and Allow Lists + +When you migrate data using TiDB Data Migration (DM), you can configure the block and allow lists to filter or only migrate all operations of some databases or some tables. + +## Configure the block and allow lists + +In the task configuration file, add the following configuration: + +```yaml +block-allow-list: # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2. + rule-1: + do-dbs: ["test*"] # Starting with characters other than "~" indicates that it is a wildcard; + # v1.0.5 or later versions support the regular expression rules. + do-tables: + - db-name: "test[123]" # Matches test1, test2, and test3. + tbl-name: "t[1-5]" # Matches t1, t2, t3, t4, and t5. + - db-name: "test" + tbl-name: "t" + rule-2: + do-dbs: ["~^test.*"] # Starting with "~" indicates that it is a regular expression. + ignore-dbs: ["mysql"] + do-tables: + - db-name: "~^test.*" + tbl-name: "~^t.*" + - db-name: "test" + tbl-name: "t" + ignore-tables: + - db-name: "test" + tbl-name: "log" +``` + +In simple scenarios, it is recommended that you use the wildcard for matching schemas and tables. However, note the following version differences: + +- For DM v1.0.5 or later versions, the block and allow lists support the [wildcard match](https://en.wikipedia.org/wiki/Glob_(programming)#Syntax), but there can be **only one** `*` in the wildcard expression, and `*` **must be placed at the end**. + +- For DM versions earlier than v1.0.5, the block and allow lists only support regular expression matching. + +## Parameter descriptions + +- `do-dbs`: allow lists of the schemas to be migrated, similar to [`replicate-do-db`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-do-db) in MySQL. +- `ignore-dbs`: block lists of the schemas to be migrated, similar to [`replicate-ignore-db`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-ignore-db) in MySQL. +- `do-tables`: allow lists of the tables to be migrated, similar to [`replicate-do-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-do-table) in MySQL. Both `db-name` and `tbl-name` must be specified. +- `ignore-tables`: block lists of the tables to be migrated, similar to [`replicate-ignore-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-ignore-table) in MySQL. Both `db-name` and `tbl-name` must be specified. + +If a value of the above parameters starts with the `~` character, the subsequent characters of this value are treated as a [regular expression](https://golang.org/pkg/regexp/syntax/#hdr-syntax). You can use this parameter to match schema or table names. + +## Filtering process + +- The filtering rules corresponding to `do-dbs` and `ignore-dbs` are similar to the [Evaluation of Database-Level Replication and Binary Logging Options](https://dev.mysql.com/doc/refman/5.7/en/replication-rules-db-options.html) in MySQL. +- The filtering rules corresponding to `do-tables` and `ignore-tables` are similar to the [Evaluation of Table-Level Replication Options](https://dev.mysql.com/doc/refman/5.7/en/replication-rules-table-options.html) in MySQL. + +> **Note:** +> +> In DM and in MySQL, the block and allow lists filtering rules are different in the following ways: +> +> - In MySQL, [`replicate-wild-do-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-wild-do-table) and [`replicate-wild-ignore-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-wild-ignore-table) support wildcard characters. In DM, some parameter values directly supports regular expressions that start with the `~` character. +> - DM currently only supports binlogs in the `ROW` format, and does not support those in the `STATEMENT` or `MIXED` format. Therefore, the filtering rules in DM correspond to those in the `ROW` format in MySQL. +> - MySQL determines a DDL statement only by the database name explicitly specified in the `USE` section of the statement. DM determines a statement first based on the database name section in the DDL statement. If the DDL statement does not contain such a section, DM determines the statement by the `USE` section. Suppose that the SQL statement to be determined is `USE test_db_2; CREATE TABLE test_db_1.test_table (c1 INT PRIMARY KEY)`; that `replicate-do-db=test_db_1` is configured in MySQL and `do-dbs: ["test_db_1"]` is configured in DM. Then this rule only applies to DM and not to MySQL. + +The filtering process of a `test`.`t` table is as follows: + +1. Filter at the **schema** level: + + - If `do-dbs` is not empty, check whether a matched schema exists in `do-dbs`. + + - If yes, continue to filter at the **table** level. + - If not, filter `test`.`t`. + + - If `do-dbs` is empty and `ignore-dbs` is not empty, check whether a matched schema exits in `ignore-dbs`. + + - If yes, filter `test`.`t`. + - If not, continue to filter at the **table** level. + + - If both `do-dbs` and `ignore-dbs` are empty, continue to filter at the **table** level. + +2. Filter at the **table** level: + + 1. If `do-tables` is not empty, check whether a matched table exists in `do-tables`. + + - If yes, migrate `test`.`t`. + - If not, filter `test`.`t`. + + 2. If `ignore-tables` is not empty, check whether a matched table exists in `ignore-tables`. + + - If yes, filter `test`.`t`. + - If not, migrate `test`.`t`. + + 3. If both `do-tables` and `ignore-tables` are empty, migrate `test`.`t`. + +> **Note:** +> +> To check whether the schema `test` should be filtered, you only need to filter at the schema level. + +## Usage examples + +Assume that the upstream MySQL instances include the following tables: + +``` +`logs`.`messages_2016` +`logs`.`messages_2017` +`logs`.`messages_2018` +`forum`.`users` +`forum`.`messages` +`forum_backup_2016`.`messages` +`forum_backup_2017`.`messages` +`forum_backup_2018`.`messages` +``` + +The configuration is as follows: + +```yaml +block-allow-list: # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2. + bw-rule: + do-dbs: ["forum_backup_2018", "forum"] + ignore-dbs: ["~^forum_backup_"] + do-tables: + - db-name: "logs" + tbl-name: "~_2018$" + - db-name: "~^forum.*" +​ tbl-name: "messages" + ignore-tables: + - db-name: "~.*" +​ tbl-name: "^messages.*" +``` + +After applying the `bw-rule` rule: + +| Table | Whether to filter | Why filter | +|:----|:----|:--------------| +| `logs`.`messages_2016` | Yes | The schema `logs` fails to match any `do-dbs`. | +| `logs`.`messages_2017` | Yes | The schema `logs` fails to match any `do-dbs`. | +| `logs`.`messages_2018` | Yes | The schema `logs` fails to match any `do-dbs`. | +| `forum_backup_2016`.`messages` | Yes | The schema `forum_backup_2016` fails to match any `do-dbs`. | +| `forum_backup_2017`.`messages` | Yes | The schema `forum_backup_2017` fails to match any `do-dbs`. | +| `forum`.`users` | Yes | 1. The schema `forum` matches `do-dbs` and continues to filter at the table level.
2. The schema and table fail to match any of `do-tables` and `ignore-tables` and `do-tables` is not empty. | +| `forum`.`messages` | No | 1. The schema `forum` matches `do-dbs` and continues to filter at the table level.
2. The table `messages` is in the `db-name: "~^forum.*",tbl-name: "messages"` of `do-tables`. | +| `forum_backup_2018`.`messages` | No | 1. The schema `forum_backup_2018` matches `do-dbs` and continues to filter at the table level.
2. The schema and table match the `db-name: "~^forum.*",tbl-name: "messages"` of `do-tables`. | diff --git a/dm/dm-faq.md b/dm/dm-faq.md index 2ed0424b03628..792148ff128ef 100644 --- a/dm/dm-faq.md +++ b/dm/dm-faq.md @@ -186,7 +186,7 @@ Sometimes, the error message contains the `parse statement` information, for exa if the DDL is not needed, you can use a filter rule with \"*\" schema-pattern to ignore it.\n\t : parse statement: line 1 column 11 near \"EVENT `event_del_big_table` \r\nDISABLE\" %!!(MISSING)(EXTRA string=ALTER EVENT `event_del_big_table` \r\nDISABLE ``` -The reason for this type of error is that the TiDB parser cannot parse DDL statements sent by the upstream, such as `ALTER EVENT`, so `sql-skip` does not take effect as expected. You can add [binlog event filters](/dm/dm-key-features.md#binlog-event-filter) in the configuration file to filter those statements and set `schema-pattern: "*"`. Starting from DM v2.0.1, DM pre-filters statements related to `EVENT`. +The reason for this type of error is that the TiDB parser cannot parse DDL statements sent by the upstream, such as `ALTER EVENT`, so `sql-skip` does not take effect as expected. You can add [binlog event filters](/dm/dm-binlog-event-filter.md) in the configuration file to filter those statements and set `schema-pattern: "*"`. Starting from DM v2.0.1, DM pre-filters statements related to `EVENT`. Since DM v6.0, `binlog` replaces `sql-skip` and `handle-error`. You can use the `binlog` command instead to avoid this issue. diff --git a/dm/dm-glossary.md b/dm/dm-glossary.md index b2e47d5c0bf30..708c547e808ba 100644 --- a/dm/dm-glossary.md +++ b/dm/dm-glossary.md @@ -20,7 +20,7 @@ Binlog events are information about data modification made to a MySQL or MariaDB ### Binlog event filter -[Binlog event filter](/dm/dm-key-features.md#binlog-event-filter) is a more fine-grained filtering feature than the block and allow lists filtering rule. Refer to [binlog event filter](/dm/dm-key-features.md#binlog-event-filter) for details. +[Binlog event filter](/dm/dm-binlog-event-filter.md) is a more fine-grained filtering feature than the block and allow lists filtering rule. Refer to [binlog event filter](/dm/dm-binlog-event-filter.md) for details. ### Binlog position @@ -32,7 +32,7 @@ Binlog replication processing unit is the processing unit used in DM-worker to r ### Block & allow table list -Block & allow table list is the feature that filters or only migrates all operations of some databases or some tables. Refer to [block & allow table lists](/dm/dm-key-features.md#block-and-allow-table-lists) for details. This feature is similar to [MySQL Replication Filtering](https://dev.mysql.com/doc/refman/5.6/en/replication-rules.html) and [MariaDB Replication Filters](https://mariadb.com/kb/en/replication-filters/). +Block & allow table list is the feature that filters or only migrates all operations of some databases or some tables. Refer to [block & allow table lists](/dm/dm-block-allow-table-lists.md) for details. This feature is similar to [MySQL Replication Filtering](https://dev.mysql.com/doc/refman/5.6/en/replication-rules.html) and [MariaDB Replication Filters](https://mariadb.com/kb/en/replication-filters/). ## C @@ -126,7 +126,7 @@ The subtask status is the status of a data migration subtask. The current status ### Table routing -The table routing feature enables DM to migrate a certain table of the upstream MySQL or MariaDB instance to the specified table in the downstream, which can be used to merge and migrate sharded tables. Refer to [table routing](/dm/dm-key-features.md#table-routing) for details. +The table routing feature enables DM to migrate a certain table of the upstream MySQL or MariaDB instance to the specified table in the downstream, which can be used to merge and migrate sharded tables. Refer to [table routing](/dm/dm-table-routing.md) for details. ### Task diff --git a/dm/dm-handle-performance-issues.md b/dm/dm-handle-performance-issues.md index 1088dd95ea5e3..b7fa4d8aecebc 100644 --- a/dm/dm-handle-performance-issues.md +++ b/dm/dm-handle-performance-issues.md @@ -72,7 +72,7 @@ The Binlog replication unit decides whether to read the binlog event from the up ### binlog event conversion -The Binlog replication unit constructs DML, parses DDL, and performs [table router](/dm/dm-key-features.md#table-routing) conversion from binlog event data. The related metric is `transform binlog event duration`. +The Binlog replication unit constructs DML, parses DDL, and performs [table router](/dm/dm-table-routing.md) conversion from binlog event data. The related metric is `transform binlog event duration`. The duration is mainly affected by the write operations upstream. Take the `INSERT INTO` statement as an example, the time consumed to convert a single `VALUES` greatly differs from that to convert a lot of `VALUES`. The time consumed might range from tens of microseconds to hundreds of microseconds. However, usually this is not a bottleneck of the system. diff --git a/dm/dm-key-features.md b/dm/dm-key-features.md deleted file mode 100644 index 702c908ca66a5..0000000000000 --- a/dm/dm-key-features.md +++ /dev/null @@ -1,577 +0,0 @@ ---- -title: Key Features of TiDB Data Migration -summary: Learn about the key features of DM and appropriate parameter configurations. -aliases: ['/docs/tidb-data-migration/dev/feature-overview/'] ---- - -# Key Features of TiDB Data Migration - -This document describes the data migration features provided by TiDB Data Migration (DM) and introduces appropriate parameter configurations. - -For different DM versions, pay attention to the different match rules of schema or table names in the table routing, block & allow lists, and binlog event filter features: - -+ For DM v1.0.5 or later versions, all the above features support the [wildcard match](https://en.wikipedia.org/wiki/Glob_(programming)#Syntax). For all versions of DM, note that there can be **only one** `*` in the wildcard expression, and `*` **must be placed at the end**. -+ For DM versions earlier than v1.0.5, table routing and binlog event filter support the wildcard but do not support the `[...]` and `[!...]` expressions. The block & allow lists only supports the regular expression. - -It is recommended that you use the wildcard for matching in simple scenarios. - -## Table routing - -The table routing feature enables DM to migrate a certain table of the upstream MySQL or MariaDB instance to the specified table in the downstream. - -> **Note:** -> -> - Configuring multiple different routing rules for a single table is not supported. -> - The match rule of schema needs to be configured separately, which is used to migrate `CREATE/DROP SCHEMA xx`, as shown in `rule-2` of the [parameter configuration](#parameter-configuration). - -### Parameter configuration - -```yaml -routes: - rule-1: - schema-pattern: "test_*" - table-pattern: "t_*" - target-schema: "test" - target-table: "t" - # extract-table, extract-schema, and extract-source are optional and are required only when you need to extract information about sharded tables, sharded schemas, and source datatabase information. - extract-table: - table-regexp: "t_(.*)" - target-column: "c_table" - extract-schema: - schema-regexp: "test_(.*)" - target-column: "c_schema" - extract-source: - source-regexp: "(.*)" - target-column: "c_source" - rule-2: - schema-pattern: "test_*" - target-schema: "test" -``` - -### Parameter explanation - -- DM migrates the upstream MySQL or MariaDB instance tables that match the [`schema-pattern`/`table-pattern` rule provided by Table selector](/dm/table-selector.md) to the downstream `target-schema`/`target-table`. -- For sharded tables that match the `schema-pattern`/`table-pattern` rules, DM extracts the table name by using the `extract-table`.`table-regexp` regular expression, the schema name by using the `extract-schema`.`schema-regexp` regular expression, and source information by using the `extract-source`.`source-regexp` regular expression. Then DM writes the extracted information to the corresponding `target-column` in the merged table in the downstream. - -### Usage examples - -This section shows the usage examples in different scenarios. - -#### Merge sharded schemas and tables - -Assuming in the scenario of sharded schemas and tables, you want to migrate the `test_{1,2,3...}`.`t_{1,2,3...}` tables in two upstream MySQL instances to the `test`.`t` table in the downstream TiDB instance. - -To migrate the upstream instances to the downstream `test`.`t`, you must create the following routing rules: - -- `rule-1` is used to migrate DML or DDL statements of the table that matches `schema-pattern: "test_*"` and `table-pattern: "t_*"` to the downstream `test`.`t`. -- `rule-2` is used to migrate DDL statements of the schema that matches `schema-pattern: "test_*"`, such as `CREATE/DROP SCHEMA xx`. - -> **Note:** -> -> - If the downstream `schema: test` already exists and is not to be deleted, you can omit `rule-2`. -> - If the downstream `schema: test` does not exist and only `rule-1` is configured, then it reports the `schema test doesn't exist` error during migration. - -```yaml - rule-1: - schema-pattern: "test_*" - table-pattern: "t_*" - target-schema: "test" - target-table: "t" - rule-2: - schema-pattern: "test_*" - target-schema: "test" -``` - -#### Extract table, schema, and source information and write into the merged table - -Assuming in the scenario of sharded schemas and tables, you want to migrate the `test_{1,2,3...}`.`t_{1,2,3...}` tables in two upstream MySQL instances to the `test`.`t` table in the downstream TiDB instance. At the same time, you want to extract the source information of the sharded tables and write it to the downstream merged table. - -To migrate the upstream instances to the downstream `test`.`t`, you must create routing rules similar to the previous section [Merge sharded schemas and tables](#merge-sharded-schemas-and-tables). In addtion, you need to add the `extract-table`, `extract-schema`, and `extract-source` configurations: - -- `extract-table`: For a sharded table matching `schema-pattern` and `table-pattern`, DM extracts the sharded table name by using `table-regexp` and writes the name suffix without the `t_` part to `target-column` of the merged table, that is, the `c_table` column. -- `extract-schema`: For a sharded schema matching `schema-pattern` and `table-pattern`, DM extracts the sharded schema name by using `schema-regexp` and writes the name suffix without the `test_` part to `target-column` of the merged table, that is, the `c_schema` column. -- `extract-source`: For a sharded table matching `schema-pattern` and `table-pattern`, DM writes the source instance information to the `target-column` of the merged table, that is, the `c_source` column. - -```yaml - rule-1: - schema-pattern: "test_*" - table-pattern: "t_*" - target-schema: "test" - target-table: "t" - extract-table: - table-regexp: "t_(.*)" - target-column: "c_table" - extract-schema: - schema-regexp: "test_(.*)" - target-column: "c_schema" - extract-source: - source-regexp: "(.*)" - target-column: "c_source" - rule-2: - schema-pattern: "test_*" - target-schema: "test" -``` - -To extract the source information of upstream sharded tables to the merged table in the downstream, you **must manually create a merged table in the downstream before starting the migration**. The merged table must contain the three `target-columns` (`c_table`, `c_schema`, and `c_source`) used for specifying the source information. In addition, these columns **must be the last columns and be [string types](/data-type-string.md)**. - -```sql -CREATE TABLE `test`.`t` ( - a int(11) PRIMARY KEY, - c_table varchar(10) DEFAULT NULL, - c_schema varchar(10) DEFAULT NULL, - c_source varchar(10) DEFAULT NULL -); -``` - -Assume that the upstream has the following two data sources: - -Data source `mysql-01`: - -```sql -mysql> select * from test_11.t_1; -+---+ -| a | -+---+ -| 1 | -+---+ -mysql> select * from test_11.t_2; -+---+ -| a | -+---+ -| 2 | -+---+ -mysql> select * from test_12.t_1; -+---+ -| a | -+---+ -| 3 | -+---+ -``` - -Data source `mysql-02`: - -```sql -mysql> select * from test_13.t_3; -+---+ -| a | -+---+ -| 4 | -+---+ -``` - -After migration using DM, data in the merged table will be as follows: - -```sql -mysql> select * from test.t; -+---+---------+----------+----------+ -| a | c_table | c_schema | c_source | -+---+---------+----------+----------+ -| 1 | 1 | 11 | mysql-01 | -| 2 | 2 | 11 | mysql-01 | -| 3 | 1 | 12 | mysql-01 | -| 4 | 3 | 13 | mysql-02 | -+---+---------+----------+----------+ -``` - -##### Incorrect examples of creating merged tables - -> **Note:** -> -> If any of the following errors occur, source information of sharded tables and schemas might fail to be written to the merged table. - -- `c-table` is not in the last three columns: - -```sql -CREATE TABLE `test`.`t` ( - c_table varchar(10) DEFAULT NULL, - a int(11) PRIMARY KEY, - c_schema varchar(10) DEFAULT NULL, - c_source varchar(10) DEFAULT NULL -); -``` - -- `c-source` is absent: - -```sql -CREATE TABLE `test`.`t` ( - a int(11) PRIMARY KEY, - c_table varchar(10) DEFAULT NULL, - c_schema varchar(10) DEFAULT NULL, -); -``` - -- `c_schema` is not a string type: - -```sql -CREATE TABLE `test`.`t` ( - a int(11) PRIMARY KEY, - c_table varchar(10) DEFAULT NULL, - c_schema int(11) DEFAULT NULL, - c_source varchar(10) DEFAULT NULL, -); -``` - -#### Merge sharded schemas - -Assuming in the scenario of sharded schemas, you want to migrate the `test_{1,2,3...}`.`t_{1,2,3...}` tables in the two upstream MySQL instances to the `test`.`t_{1,2,3...}` tables in the downstream TiDB instance. - -To migrate the upstream schemas to the downstream `test`.`t_[1,2,3]`, you only need to create one routing rule. - -```yaml - rule-1: - schema-pattern: "test_*" - target-schema: "test" -``` - -#### Incorrect table routing - -Assuming that the following two routing rules are configured and `test_1_bak`.`t_1_bak` matches both `rule-1` and `rule-2`, an error is reported because the table routing configuration violates the number limitation. - -```yaml - rule-1: - schema-pattern: "test_*" - table-pattern: "t_*" - target-schema: "test" - target-table: "t" - rule-2: - schema-pattern: "test_1_bak" - table-pattern: "t_1_bak" - target-schema: "test" - target-table: "t_bak" -``` - -## Block and allow table lists - -The block and allow lists filtering rule of the upstream database instance tables is similar to MySQL replication-rules-db/tables, which can be used to filter or only migrate all operations of some databases or some tables. - -### Parameter configuration - -```yaml -block-allow-list: # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2. - rule-1: - do-dbs: ["test*"] # Starting with characters other than "~" indicates that it is a wildcard; - # v1.0.5 or later versions support the regular expression rules. - do-tables: - - db-name: "test[123]" # Matches test1, test2, and test3. - tbl-name: "t[1-5]" # Matches t1, t2, t3, t4, and t5. - - db-name: "test" - tbl-name: "t" - rule-2: - do-dbs: ["~^test.*"] # Starting with "~" indicates that it is a regular expression. - ignore-dbs: ["mysql"] - do-tables: - - db-name: "~^test.*" - tbl-name: "~^t.*" - - db-name: "test" - tbl-name: "t" - ignore-tables: - - db-name: "test" - tbl-name: "log" -``` - -### Parameter explanation - -- `do-dbs`: allow lists of the schemas to be migrated, similar to [`replicate-do-db`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-do-db) in MySQL -- `ignore-dbs`: block lists of the schemas to be migrated, similar to [`replicate-ignore-db`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-ignore-db) in MySQL -- `do-tables`: allow lists of the tables to be migrated, similar to [`replicate-do-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-do-table) in MySQL. Both `db-name` and `tbl-name` must be specified -- `ignore-tables`: block lists of the tables to be migrated, similar to [`replicate-ignore-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-ignore-table) in MySQL. Both `db-name` and `tbl-name` must be specified - -If a value of the above parameters starts with the `~` character, the subsequent characters of this value are treated as a [regular expression](https://golang.org/pkg/regexp/syntax/#hdr-syntax). You can use this parameter to match schema or table names. - -### Filtering process - -The filtering rules corresponding to `do-dbs` and `ignore-dbs` are similar to the [Evaluation of Database-Level Replication and Binary Logging Options](https://dev.mysql.com/doc/refman/5.7/en/replication-rules-db-options.html) in MySQL. The filtering rules corresponding to `do-tables` and `ignore-tables` are similar to the [Evaluation of Table-Level Replication Options](https://dev.mysql.com/doc/refman/5.7/en/replication-rules-table-options.html) in MySQL. - -> **Note:** -> -> In DM and in MySQL, the allow and block lists filtering rules are different in the following ways: -> -> - In MySQL, [`replicate-wild-do-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-wild-do-table) and [`replicate-wild-ignore-table`](https://dev.mysql.com/doc/refman/5.7/en/replication-options-replica.html#option_mysqld_replicate-wild-ignore-table) support wildcard characters. In DM, some parameter values directly supports regular expressions that start with the `~` character. -> - DM currently only supports binlogs in the `ROW` format, and does not support those in the `STATEMENT` or `MIXED` format. Therefore, the filtering rules in DM correspond to those in the `ROW` format in MySQL. -> - MySQL determines a DDL statement only by the database name explicitly specified in the `USE` section of the statement. DM determines a statement first based on the database name section in the DDL statement. If the DDL statement does not contain such a section, DM determines the statement by the `USE` section. Suppose that the SQL statement to be determined is `USE test_db_2; CREATE TABLE test_db_1.test_table (c1 INT PRIMARY KEY)`; that `replicate-do-db=test_db_1` is configured in MySQL and `do-dbs: ["test_db_1"]` is configured in DM. Then this rule only applies to DM and not to MySQL. - -The filtering process is as follows: - -1. Filter at the schema level: - - - If `do-dbs` is not empty, judge whether a matched schema exists in `do-dbs`. - - - If yes, continue to filter at the table level. - - If not, filter `test`.`t`. - - - If `do-dbs` is empty and `ignore-dbs` is not empty, judge whether a matched schema exits in `ignore-dbs`. - - - If yes, filter `test`.`t`. - - If not, continue to filter at the table level. - - - If both `do-dbs` and `ignore-dbs` are empty, continue to filter at the table level. - -2. Filter at the table level: - - 1. If `do-tables` is not empty, judge whether a matched table exists in `do-tables`. - - - If yes, migrate `test`.`t`. - - If not, filter `test`.`t`. - - 2. If `ignore-tables` is not empty, judge whether a matched table exists in `ignore-tables`. - - - If yes, filter `test`.`t`. - - If not, migrate `test`.`t`. - - 3. If both `do-tables` and `ignore-tables` are empty, migrate `test`.`t`. - -> **Note:** -> -> To judge whether the schema `test` should be filtered, you only need to filter at the schema level. - -### Usage example - -Assume that the upstream MySQL instances include the following tables: - -``` -`logs`.`messages_2016` -`logs`.`messages_2017` -`logs`.`messages_2018` -`forum`.`users` -`forum`.`messages` -`forum_backup_2016`.`messages` -`forum_backup_2017`.`messages` -`forum_backup_2018`.`messages` -``` - -The configuration is as follows: - -```yaml -block-allow-list: # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2. - bw-rule: - do-dbs: ["forum_backup_2018", "forum"] - ignore-dbs: ["~^forum_backup_"] - do-tables: - - db-name: "logs" - tbl-name: "~_2018$" - - db-name: "~^forum.*" -​ tbl-name: "messages" - ignore-tables: - - db-name: "~.*" -​ tbl-name: "^messages.*" -``` - -After using the `bw-rule` rule: - -| Table | Whether to filter | Why filter | -|:----|:----|:--------------| -| `logs`.`messages_2016` | Yes | The schema `logs` fails to match any `do-dbs`. | -| `logs`.`messages_2017` | Yes | The schema `logs` fails to match any `do-dbs`. | -| `logs`.`messages_2018` | Yes | The schema `logs` fails to match any `do-dbs`. | -| `forum_backup_2016`.`messages` | Yes | The schema `forum_backup_2016` fails to match any `do-dbs`. | -| `forum_backup_2017`.`messages` | Yes | The schema `forum_backup_2017` fails to match any `do-dbs`. | -| `forum`.`users` | Yes | 1. The schema `forum` matches `do-dbs` and continues to filter at the table level.
2. The schema and table fail to match any of `do-tables` and `ignore-tables` and `do-tables` is not empty. | -| `forum`.`messages` | No | 1. The schema `forum` matches `do-dbs` and continues to filter at the table level.
2. The table `messages` is in the `db-name: "~^forum.*",tbl-name: "messages"` of `do-tables`. | -| `forum_backup_2018`.`messages` | No | 1. The schema `forum_backup_2018` matches `do-dbs` and continues to filter at the table level.
2. The schema and table match the `db-name: "~^forum.*",tbl-name: "messages"` of `do-tables`. | - -## Binlog event filter - -Binlog event filter is a more fine-grained filtering rule than the block and allow lists filtering rule. You can use statements like `INSERT` or `TRUNCATE TABLE` to specify the binlog events of `schema/table` that you need to migrate or filter out. - -> **Note:** -> -> - If the same table matches multiple rules, these rules are applied in order and the block list has priority over the allow list. This means if both the `Ignore` and `Do` rules are applied to a table, the `Ignore` rule takes effect. -> - Starting from DM v2.0.2, you can configure binlog event filters in the source configuration file. For details, see [Upstream Database Configuration File](/dm/dm-source-configuration-file.md). - -### Parameter configuration - -```yaml -filters: - rule-1: - schema-pattern: "test_*" - ​table-pattern: "t_*" - ​events: ["truncate table", "drop table"] - sql-pattern: ["^DROP\\s+PROCEDURE", "^CREATE\\s+PROCEDURE"] - ​action: Ignore -``` - -### Parameter explanation - -- [`schema-pattern`/`table-pattern`](/dm/table-selector.md): the binlog events or DDL SQL statements of upstream MySQL or MariaDB instance tables that match `schema-pattern`/`table-pattern` are filtered by the rules below. - -- `events`: the binlog event array. You can only select one or more `Event`s from the following table: - - | Events | Type | Description | - | --------------- | ---- | ----------------------------- | - | `all` | | Includes all the events below | - | `all dml` | | Includes all DML events below | - | `all ddl` | | Includes all DDL events below | - | `none` | | Includes none of the events below | - | `none ddl` | | Includes none of the DDL events below | - | `none dml` | | Includes none of the DML events below | - | `insert` | DML | The `INSERT` DML event | - | `update` | DML | The `UPDATE` DML event | - | `delete` | DML | The `DELETE` DML event | - | `create database` | DDL | The `CREATE DATABASE` DDL event | - | `drop database` | DDL | The `DROP DATABASE` DDL event | - | `create table` | DDL | The `CREATE TABLE` DDL event | - | `create index` | DDL | The `CREATE INDEX` DDL event | - | `drop table` | DDL | The `DROP TABLE` DDL event | - | `truncate table` | DDL | The `TRUNCATE TABLE` DDL event | - | `rename table` | DDL | The `RENAME TABLE` DDL event | - | `drop index` | DDL | The `DROP INDEX` DDL event | - | `alter table` | DDL | The `ALTER TABLE` DDL event | - -- `sql-pattern`: it is used to filter specified DDL SQL statements. The matching rule supports using a regular expression. For example, `"^DROP\\s+PROCEDURE"`. - -- `action`: the string (`Do`/`Ignore`). Based on the following rules, it judges whether to filter. If either of the two rules is satisfied, the binlog is filtered; otherwise, the binlog is not filtered. - - - `Do`: the allow list. The binlog is filtered in either of the following two conditions: - - The type of the event is not in the `event` list of the rule. - - The SQL statement of the event cannot be matched by `sql-pattern` of the rule. - - `Ignore`: the block list. The binlog is filtered in either of the following two conditions: - - The type of the event is in the `event` list of the rule. - - The SQL statement of the event can be matched by `sql-pattern` of the rule. - -### Usage examples - -This section shows the usage examples in the scenario of sharding (sharded schemas and tables). - -#### Filter all sharding deletion operations - -To filter out all deletion operations, configure the following two filtering rules: - -- `filter-table-rule` filters out the `truncate table`, `drop table` and `delete statement` operations of all tables that match the `test_*`.`t_*` pattern. -- `filter-schema-rule` filters out the `drop database` operation of all schemas that match the `test_*` pattern. - -```yaml -filters: - filter-table-rule: - schema-pattern: "test_*" - table-pattern: "t_*" - events: ["truncate table", "drop table", "delete"] - action: Ignore - filter-schema-rule: - schema-pattern: "test_*" - events: ["drop database"] - action: Ignore -``` - -#### Only migrate sharding DML statements - -To only migrate sharding DML statements, configure the following two filtering rules: - -- `do-table-rule` only migrates the `create table`, `insert`, `update` and `delete` statements of all tables that match the `test_*`.`t_*` pattern. -- `do-schema-rule` only migrates the `create database` statement of all schemas that match the `test_*` pattern. - -> **Note:** -> -> The reason why the `create database/table` statement is migrated is that you can migrate DML statements only after the schema and table are created. - -```yaml -filters: - do-table-rule: - schema-pattern: "test_*" - table-pattern: "t_*" - events: ["create table", "all dml"] - action: Do - do-schema-rule: - schema-pattern: "test_*" - events: ["create database"] - action: Do -``` - -#### Filter out the SQL statements that TiDB does not support - -To filter out the `PROCEDURE` statements that TiDB does not support, configure the following `filter-procedure-rule`: - -```yaml -filters: - filter-procedure-rule: - schema-pattern: "test_*" - table-pattern: "t_*" - sql-pattern: ["^DROP\\s+PROCEDURE", "^CREATE\\s+PROCEDURE"] - action: Ignore -``` - -`filter-procedure-rule` filters out the `^CREATE\\s+PROCEDURE` and `^DROP\\s+PROCEDURE` statements of all tables that match the `test_*`.`t_*` pattern. - -#### Filter out the SQL statements that the TiDB parser does not support - -For the SQL statements that the TiDB parser does not support, DM cannot parse them and get the `schema`/`table` information. So you must use the global filtering rule: `schema-pattern: "*"`. - -> **Note:** -> -> To avoid filtering out data that need to be migrated, you must configure the global filtering rule as strictly as possible. - -To filter out the `PARTITION` statements that the TiDB parser (of some version) does not support, configure the following filtering rule: - -```yaml -filters: - filter-partition-rule: - schema-pattern: "*" - sql-pattern: ["ALTER\\s+TABLE[\\s\\S]*ADD\\s+PARTITION", "ALTER\\s+TABLE[\\s\\S]*DROP\\s+PARTITION"] - action: Ignore -``` - -## Online DDL tools - -In the MySQL ecosystem, tools such as gh-ost and pt-osc are widely used. DM provides supports for these tools to avoid migrating unnecessary intermediate data. - -### Restrictions - -- DM only supports gh-ost and pt-osc. -- When `online-ddl` is enabled, the checkpoint corresponding to incremental replication should not be in the process of online DDL execution. For example, if an upstream online DDL operation starts at `position-A` and ends at `position-B` of the binlog, the starting point of incremental replication should be earlier than `position-A` or later than `position-B`; otherwise, an error occurs. For details, refer to [FAQ](/dm/dm-faq.md#how-to-handle-the-error-returned-by-the-ddl-operation-related-to-the-gh-ost-table-after-online-ddl-scheme-gh-ost-is-set). - -### Parameter configuration - - -
- -In v2.0.5 and later versions, you need to use the `online-ddl` configuration item in the `task` configuration file. - -- If the upstream MySQL/MariaDB (at the same time) uses the gh-ost or pt-osc tool, set `online-ddl` to `true` in the task configuration file: - -```yml -online-ddl: true -``` - -> **Note:** -> -> Since v2.0.5, `online-ddl-scheme` has been deprecated, so you need to use `online-ddl` instead of `online-ddl-scheme`. That means that setting `online-ddl: true` overwrites `online-ddl-scheme`, and setting `online-ddl-scheme: "pt"` or `online-ddl-scheme: "gh-ost"` is converted to `online-ddl: true`. - -
- -
- -Before v2.0.5 (not including v2.0.5), you need to use the `online-ddl-scheme` configuration item in the `task` configuration file. - -- If the upstream MySQL/MariaDB uses the gh-ost tool, set it in the task configuration file: - -```yml -online-ddl-scheme: "gh-ost" -``` - -- If the upstream MySQL/MariaDB uses the pt tool, set it in the task configuration file: - -```yml -online-ddl-scheme: "pt" -``` - -
- - -## Shard merge - -DM supports merging the DML and DDL data in the upstream MySQL/MariaDB sharded tables and migrating the merged data to the downstream TiDB tables. - -### Restrictions - -Currently, the shard merge feature is supported only in limited scenarios. For details, refer to [Sharding DDL usage Restrictions in the pessimistic mode](/dm/feature-shard-merge-pessimistic.md#restrictions) and [Sharding DDL usage Restrictions in the optimistic mode](/dm/feature-shard-merge-optimistic.md#restrictions). - -### Parameter configuration - -Set `shard-mode` to `pessimistic` in the task configuration file: - -``` -shard-mode: "pessimistic" # The shard merge mode. Optional modes are ""/"pessimistic"/"optimistic". The "" mode is used by default which means sharding DDL merge is disabled. If the task is a shard merge task, set it to the "pessimistic" mode. After getting a deep understanding of the principles and restrictions of the "optimistic" mode, you can set it to the "optimistic" mode. -``` - -### Handle sharding DDL locks manually - -In some abnormal scenarios, you need to [handle sharding DDL Locks manually](/dm/manually-handling-sharding-ddl-locks.md). diff --git a/dm/dm-online-ddl-tool-support.md b/dm/dm-online-ddl-tool-support.md new file mode 100644 index 0000000000000..3ed9816841b53 --- /dev/null +++ b/dm/dm-online-ddl-tool-support.md @@ -0,0 +1,55 @@ +--- +title: TiDB Data Migration Support for Online DDL Tools +summary: Learn about the support for common online DDL tools, usage, and precautions in DM. +--- + +# TiDB Data Migration Support for Online DDL Tools + +In the MySQL ecosystem, tools such as gh-ost and pt-osc are widely used. TiDB Data Migration (DM) provides supports for these tools to avoid migrating unnecessary intermediate data. + +This document introduces the support for common online DDL tools, usage, and precautions in DM. + +For the working principles and implementation methods of DM for online DDL tools, refer to [online-ddl](/dm/feature-online-ddl.md). + +## Restrictions + +- DM only supports gh-ost and pt-osc. +- When `online-ddl` is enabled, the checkpoint corresponding to incremental replication should not be in the process of online DDL execution. For example, if an upstream online DDL operation starts at `position-A` and ends at `position-B` of the binlog, the starting point of incremental replication should be earlier than `position-A` or later than `position-B`; otherwise, an error occurs. For details, refer to [FAQ](/dm/dm-faq.md#how-to-handle-the-error-returned-by-the-ddl-operation-related-to-the-gh-ost-table-after-online-ddl-scheme-gh-ost-is-set). + +## Configure parameters + + +
+ +In v2.0.5 and later versions, you need to use the `online-ddl` configuration item in the `task` configuration file. + +- If the upstream MySQL/MariaDB (at the same time) uses the gh-ost or pt-osc tool, set `online-ddl` to `true` in the task configuration file: + +```yml +online-ddl: true +``` + +> **Note:** +> +> Since v2.0.5, `online-ddl-scheme` has been deprecated, so you need to use `online-ddl` instead of `online-ddl-scheme`. That means that setting `online-ddl: true` overwrites `online-ddl-scheme`, and setting `online-ddl-scheme: "pt"` or `online-ddl-scheme: "gh-ost"` is converted to `online-ddl: true`. + +
+ +
+ +Before v2.0.5 (not including v2.0.5), you need to use the `online-ddl-scheme` configuration item in the `task` configuration file. + +- If the upstream MySQL/MariaDB uses the gh-ost tool, set it in the task configuration file: + +```yml +online-ddl-scheme: "gh-ost" +``` + +- If the upstream MySQL/MariaDB uses the pt tool, set it in the task configuration file: + +```yml +online-ddl-scheme: "pt" +``` + +
+ diff --git a/dm/dm-overview.md b/dm/dm-overview.md index 6f8f44dc4f3f1..9509ea42854b4 100644 --- a/dm/dm-overview.md +++ b/dm/dm-overview.md @@ -1,12 +1,12 @@ --- -title: Data Migration Overview +title: TiDB Data Migration Overview summary: Learn about the Data Migration tool, the architecture, the key components, and features. -aliases: ['/docs/tidb-data-migration/dev/overview/'] +aliases: ['/docs/tidb-data-migration/dev/overview/','/docs/tidb-data-migration/dev/feature-overview/','/tidb/dev/dm-key-features'] --- -# Data Migration Overview +# TiDB Data Migration Overview