From 4ad3e599c845e05d2ce005cee69a8fdec1bdb577 Mon Sep 17 00:00:00 2001 From: kennytm Date: Fri, 3 Jul 2020 15:20:17 +0800 Subject: [PATCH 1/5] cherry pick #3065 to release-3.1 Signed-off-by: ti-srebot --- TOC.md | 235 ++++++++++++++++ br/backup-and-restore-tool.md | 37 +++ table-filter.md | 252 ++++++++++++++++++ .../tidb-lightning-configuration.md | 29 ++ tidb-lightning/tidb-lightning-glossary.md | 16 +- 5 files changed, 563 insertions(+), 6 deletions(-) create mode 100644 table-filter.md diff --git a/TOC.md b/TOC.md index fa9d02c194286..4bf2e31090dca 100644 --- a/TOC.md +++ b/TOC.md @@ -92,6 +92,7 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale +<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -99,6 +100,130 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) +======= + + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) + + [Use TiDB Ansible](/scale-tidb-using-ansible.md) + + [Use TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/scale-a-tidb-cluster) + + Backup and Restore + + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) + + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) + + Use BR Tool + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + [BR storages](/br/backup-and-restore-storages.md) + + [Configure Time Zone](/configure-time-zone.md) + + [Daily Checklist](/daily-check.md) + + [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) + + [Maintain TiFlash](/tiflash/maintain-tiflash.md) + + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) + + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) ++ Monitor and Alert + + [Monitoring Framework Overview](/tidb-monitoring-framework.md) + + [Monitoring API](/tidb-monitoring-api.md) + + [Deploy Monitoring Services](/deploy-monitoring-services.md) + + [TiDB Cluster Alert Rules](/alert-rules.md) + + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) ++ Troubleshoot + + [Identify Slow Queries](/identify-slow-queries.md) + + [SQL Diagnostics](/system-tables/system-table-sql-diagnostics.md) + + [Identify Expensive Queries](/identify-expensive-queries.md) + + [Statement Summary Tables](/statement-summary-tables.md) + + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) + + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) + + [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) + + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) ++ Performance Tuning + + System Tuning + + [Operating System Tuning](/tune-operating-system.md) + + Software Tuning + + Configuration + + [Tune TiDB Memory](/configure-memory-usage.md) + + [Tune TiKV Threads](/tune-tikv-thread-performance.md) + + [Tune TiKV Memory](/tune-tikv-memory-performance.md) + + [TiKV Follower Read](/follower-read.md) + + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) + + [Coprocessor Cache](/coprocessor-cache.md) + + SQL Tuning + + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) + + SQL Optimization + + [SQL Optimization Process](/sql-optimization-concepts.md) + + Logic Optimization + + [Join Reorder](/join-reorder.md) + + Physical Optimization + + [Statistics](/statistics.md) + + Control Execution Plan + + [Optimizer Hints](/optimizer-hints.md) + + [SQL Plan Management](/sql-plan-management.md) + + [Access Tables Using `IndexMerge`](/index-merge.md) ++ Tutorials + + [Multiple Data Centers in One City Deployment](/multi-data-centers-in-one-city-deployment.md) + + [Three Data Centers in Two Cities Deployment](/three-data-centers-in-two-cities-deployment.md) + + Best Practices + + [Use TiDB](/tidb-best-practices.md) + + [Java Application Development](/best-practices/java-app-best-practices.md) + + [Use HAProxy](/best-practices/haproxy-best-practices.md) + + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) + + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) + + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) + + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) + + [Use Placement Rules](/configure-placement-rules.md) + + [Use Load Base Split](/configure-load-base-split.md) + + [Use Store Limit](/configure-store-limit.md) ++ TiDB Ecosystem Tools + + [Overview](/ecosystem-tool-user-guide.md) + + [Use Cases](/ecosystem-tool-user-case.md) + + [Download](/download-ecosystem-tools.md) + + Backup & Restore (BR) + + [BR FAQ](/br/backup-and-restore-faq.md) + + [Use BR Tool](/br/backup-and-restore-tool.md) + + [BR Use Cases](/br/backup-and-restore-use-cases.md) + + TiDB Binlog + + [Overview](/tidb-binlog/tidb-binlog-overview.md) + + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) + + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) + + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) + + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) + + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) + + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) + + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) + + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) + + [binlogctl](/tidb-binlog/binlog-control.md) + + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) + + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) + + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) + + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) + + Troubleshoot + + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) + + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) + + [FAQ](/tidb-binlog/tidb-binlog-faq.md) + + TiDB Lightning + + [Overview](/tidb-lightning/tidb-lightning-overview.md) + + [Tutorial](/get-started-with-tidb-lightning.md) + + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) + + [Configure](/tidb-lightning/tidb-lightning-configuration.md) + + Key Features + + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) + + [Table Filter](/table-filter.md) + + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) + + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) + + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) + + [Troubleshoot](/troubleshoot-tidb-lightning.md) + + [FAQ](/tidb-lightning/tidb-lightning-faq.md) + + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) + + [TiCDC](/ticdc/ticdc-overview.md) + + sync-diff-inspector + + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) + + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) + + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) + + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) + + [Loader](/loader-overview.md) + + [Mydumper](/mydumper-overview.md) + + [Syncer](/syncer-overview.md) + + TiSpark + + [Quick Start](/get-started-with-tispark.md) + + [User Guide](/tispark-overview.md) +>>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) @@ -148,6 +273,7 @@ - [`SET`](/data-type-string.md#set-type) - [JSON Type](/data-type-json.md) + Functions and Operators +<<<<<<< HEAD - [Function and Operator Reference](/functions-and-operators/functions-and-operators-overview.md) - [Type Conversion in Expression Evaluation](/functions-and-operators/type-conversion-in-expression-evaluation.md) - [Operators](/functions-and-operators/operators.md) @@ -374,6 +500,115 @@ - [TiKV Control](/tikv-control.md) - [TiDB Control](/tidb-control.md) - [TiDB in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/) +======= + + [Overview](/functions-and-operators/functions-and-operators-overview.md) + + [Type Conversion in Expression Evaluation](/functions-and-operators/type-conversion-in-expression-evaluation.md) + + [Operators](/functions-and-operators/operators.md) + + [Control Flow Functions](/functions-and-operators/control-flow-functions.md) + + [String Functions](/functions-and-operators/string-functions.md) + + [Numeric Functions and Operators](/functions-and-operators/numeric-functions-and-operators.md) + + [Date and Time Functions](/functions-and-operators/date-and-time-functions.md) + + [Bit Functions and Operators](/functions-and-operators/bit-functions-and-operators.md) + + [Cast Functions and Operators](/functions-and-operators/cast-functions-and-operators.md) + + [Encryption and Compression Functions](/functions-and-operators/encryption-and-compression-functions.md) + + [Information Functions](/functions-and-operators/information-functions.md) + + [JSON Functions](/functions-and-operators/json-functions.md) + + [Aggregate (GROUP BY) Functions](/functions-and-operators/aggregate-group-by-functions.md) + + [Window Functions](/functions-and-operators/window-functions.md) + + [Miscellaneous Functions](/functions-and-operators/miscellaneous-functions.md) + + [Precision Math](/functions-and-operators/precision-math.md) + + [List of Expressions for Pushdown](/functions-and-operators/expressions-pushed-down.md) + + [Constraints](/constraints.md) + + [Generated Columns](/generated-columns.md) + + [SQL Mode](/sql-mode.md) + + Transactions + + [Overview](/transaction-overview.md) + + [Isolation Levels](/transaction-isolation-levels.md) + + [Optimistic Transactions](/optimistic-transaction.md) + + [Pessimistic Transactions](/pessimistic-transaction.md) + + Garbage Collection (GC) + + [Overview](/garbage-collection-overview.md) + + [Configuration](/garbage-collection-configuration.md) + + [Views](/views.md) + + [Partitioning](/partitioned-table.md) + + [Character Set and Collation](/character-set-and-collation.md) + + System Tables + + [`mysql`](/system-tables/system-table-overview.md) + + [`information_schema`](/system-tables/system-table-information-schema.md) + + sql-diagnosis + + [`cluster_info`](/system-tables/system-table-cluster-info.md) + + [`cluster_hardware`](/system-tables/system-table-cluster-hardware.md) + + [`cluster_config`](/system-tables/system-table-cluster-config.md) + + [`cluster_load`](/system-tables/system-table-cluster-load.md) + + [`cluster_systeminfo`](/system-tables/system-table-cluster-systeminfo.md) + + [`cluster_log`](/system-tables/system-table-cluster-log.md) + + [`metrics_schema`](/system-tables/system-table-metrics-schema.md) + + [`metrics_tables`](/system-tables/system-table-metrics-tables.md) + + [`metrics_summary`](/system-tables/system-table-metrics-summary.md) + + [`inspection_result`](/system-tables/system-table-inspection-result.md) + + [`inspection_summary`](/system-tables/system-table-inspection-summary.md) + + UI + + TiDB Dashboard + + [Overview](/dashboard/dashboard-intro.md) + + Maintain + + [Deploy](/dashboard/dashboard-ops-deploy.md) + + [Reverse Proxy](/dashboard/dashboard-ops-reverse-proxy.md) + + [Secure](/dashboard/dashboard-ops-security.md) + + [Access](/dashboard/dashboard-access.md) + + [Overview Page](/dashboard/dashboard-overview.md) + + [Cluster Info Page](/dashboard/dashboard-cluster-info.md) + + [Key Visualizer Page](/dashboard/dashboard-key-visualizer.md) + + SQL Statements Analysis + + [SQL Statements Page](/dashboard/dashboard-statement-list.md) + + [SQL Details Page](/dashboard/dashboard-statement-details.md) + + [Slow Queries Page](/dashboard/dashboard-slow-query.md) + + Cluster Diagnostics + + [Access Cluster Diagnostics Page](/dashboard/dashboard-diagnostics-access.md) + + [View Diagnostics Report](/dashboard/dashboard-diagnostics-report.md) + + [Use Diagnostics](/dashboard/dashboard-diagnostics-usage.md) + + [Search Logs Page](/dashboard/dashboard-log-search.md) + + [Profile Instances Page](/dashboard/dashboard-profiling.md) + + [FAQ](/dashboard/dashboard-faq.md) + + CLI + + [tikv-ctl](/tikv-control.md) + + [pd-ctl](/pd-control.md) + + [tidb-ctl](/tidb-control.md) + + [pd-recover](/pd-recover.md) + + Command Line Flags + + [tidb-server](/command-line-flags-for-tidb-configuration.md) + + [tikv-server](/command-line-flags-for-tikv-configuration.md) + + [tiflash-server](/tiflash/tiflash-command-line-flags.md) + + [pd-server](/command-line-flags-for-pd-configuration.md) + + Configuration File Parameters + + [tidb-server](/tidb-configuration-file.md) + + [tikv-server](/tikv-configuration-file.md) + + [tiflash-server](/tiflash/tiflash-configuration.md) + + [pd-server](/pd-configuration-file.md) + + System Variables + + [MySQL System Variables](/system-variables.md) + + [TiDB Specific System Variables](/tidb-specific-system-variables.md) + + Storage Engines + + TiFlash + + [Overview](/tiflash/tiflash-overview.md) + + [Use TiFlash](/tiflash/use-tiflash.md) + + TiUP + + [Documentation Guide](/tiup/tiup-documentation-guide.md) + + [Overview](/tiup/tiup-overview.md) + + [Terminology and Concepts](/tiup/tiup-terminology-and-concepts.md) + + [Manage TiUP Components](/tiup/tiup-component-management.md) + + [FAQ](/tiup/tiup-faq.md) + + [Troubleshooting Guide](/tiup/tiup-troubleshooting-guide.md) + + TiUP Components + + [tiup-playground](/tiup/tiup-playground.md) + + [tiup-cluster](/tiup/tiup-cluster.md) + + [tiup-mirror](/tiup/tiup-mirror.md) + + [tiup-bench](/tiup/tiup-bench.md) + + [Telemetry](/telemetry.md) + + [Errors Codes](/error-codes.md) + + [TiCDC Overview](/ticdc/ticdc-overview.md) + + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + + [Table Filter](/table-filter.md) +>>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) + FAQs - [TiDB FAQs](/faq/tidb-faq.md) - [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 4be2da0c029ff..f5a1a8cbe078a 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -269,6 +269,25 @@ For descriptions of other options, see [Back up all cluster data](#back-up-all-t A progress bar is displayed in the terminal during the backup operation. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data safety. +### Back up with table filter + +To back up multiple tables with more complex criteria, execute the `br backup full` command and specify the [table filters](/table-filter.md) with `--filter` or `-f`. + +**Usage example:** + +The following command backs up the data of all tables in the form `db*.tbl*` to the `/tmp/backup` path on each TiKV node and writes the `backupmeta` file to this path. + +{{< copyable "shell-regular" >}} + +```shell +br backup full \ + --pd "${PDIP}:2379" \ + --filter 'db*.tbl*' \ + --storage "local:///tmp/backup" \ + --ratelimit 120 \ + --log-file backupfull.log +``` + ### Back up data to Amazon S3 backend If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. @@ -443,6 +462,24 @@ br restore table \ In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-the-backup-data) and [Restore a database](#restore-a-database). +### Restore with table filter + +To restore multiple tables with more complex criteria, execute the `br restore full` command and specify the [table filters](/table-filter.md) with `--filter` or `-f`. + +**Usage example:** + +The following command restores a subset of tables backed up in the `/tmp/backup` path to the cluster. + +{{< copyable "shell-regular" >}} + +```shell +br restore full \ + --pd "${PDIP}:2379" \ + --filter 'db*.tbl*' \ + --storage "local:///tmp/backup" \ + --log-file restorefull.log +``` + ### Restore data from Amazon S3 backend If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. diff --git a/table-filter.md b/table-filter.md new file mode 100644 index 0000000000000..80f44ac522ec8 --- /dev/null +++ b/table-filter.md @@ -0,0 +1,252 @@ +--- +title: Table Filter +summary: Usage of table filter feature in TiDB tools. +category: reference +aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/','/docs/dev/reference/tools/tidb-lightning/table-filter/','/tidb/dev/tidb-lightning-table-filter/'] +--- + +# Table Filter + +The TiDB ecosystem tools operate on all the databases by default, but oftentimes only a subset is needed. For example, you only want to work with the schemas in the form of `foo*` and `bar*` and nothing else. + +Since TiDB 4.0, all TiDB ecosystem tools share a common filter syntax to define subsets. This document describes how to use the table filter feature. + +## Usage + +### CLI + +Table filters can be applied to the tools using multiple `-f` or `--filter` command line parameters. Each filter is in the form of `db.table`, where each part can be a wildcard (further explained in the [next section](#wildcards)). The following lists the example usage in each tool. + +* [BR](/br/backup-and-restore-tool.md): + + {{< copyable "shell-regular" >}} + + ```shell + ./br backup full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' + # ^~~~~~~~~~~~~~~~~~~~~~~ + ./br restore full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +* [Dumpling](/export-or-backup-using-dumpling.md): + + {{< copyable "shell-regular" >}} + + ```shell + ./dumpling -f 'foo*.*' -f 'bar*.*' -P 3306 -o /tmp/data/ + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +* [Lightning](/tidb-lightning/tidb-lightning-overview.md): + + {{< copyable "shell-regular" >}} + + ```shell + ./tidb-lightning -f 'foo*.*' -f 'bar*.*' -d /tmp/data/ --backend tidb + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +### TOML configuration files + +Table filters in TOML files are specified as [array of strings](https://toml.io/en/v1.0.0-rc.1#section-15). The following lists the example usage in each tool. + +* Lightning: + + ```toml + [mydumper] + filter = ['foo*.*', 'bar*.*'] + ``` + +* [TiCDC](/ticdc/ticdc-overview.md): + + ```toml + [filter] + rules = ['foo*.*', 'bar*.*'] + + [[sink.dispatchers]] + matcher = ['db1.*', 'db2.*', 'db3.*'] + dispatcher = 'ts' + ``` + +## Syntax + +### Plain table names + +Each table filter rule consists of a "schema pattern" and a "table pattern", separated by a dot (`.`). Tables whose fully-qualified name matches the rules are accepted. + +``` +db1.tbl1 +db2.tbl2 +db3.tbl3 +``` + +A plain name must only consist of valid [identifier characters](/schema-object-names.md), such as: + +* digits (`0` to `9`) +* letters (`a` to `z`, `A` to `Z`) +* `$` +* `_` +* non ASCII characters (U+0080 to U+10FFFF) + +All other ASCII characters are reserved. Some punctuations have special meanings, as described in the next section. + +### Wildcards + +Each part of the name can be a wildcard symbol described in [fnmatch(3)](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13): + +* `*` — matches zero or more characters +* `?` — matches one character +* `[a-z]` — matches one character between "a" and "z" inclusively +* `[!a-z]` — matches one character except "a" to "z". + +``` +db[0-9].tbl[0-9a-f][0-9a-f] +data.* +*.backup_* +``` + +"Character" here means a Unicode code point, such as: + +* U+00E9 (é) is 1 character. +* U+0065 U+0301 (é) are 2 characters. +* U+1F926 U+1F3FF U+200D U+2640 U+FE0F (🤦🏿‍♀️) are 5 characters. + +### File import + +To import a file as the filter rule, include an `@` at the beginning of the rule to specify the file name. The table filter parser treats each line of the imported file as additional filter rules. + +For example, if a file `config/filter.txt` has the following content: + +``` +employees.* +*.WorkOrder +``` + +the following two invocations are equivalent: + +```bash +./dumpling -f '@config/filter.txt' +./dumpling -f 'employees.*' -f '*.WorkOrder' +``` + +A filter file cannot further import another file. + +### Comments and blank lines + +Inside a filter file, leading and trailing white-spaces of every line are trimmed. Furthermore, blank lines (empty strings) are ignored. + +A leading `#` marks a comment and is ignored. `#` not at start of line is considered syntax error. + +``` +# this line is a comment +db.table # but this part is not comment and may cause error +``` + +### Exclusion + +An `!` at the beginning of the rule means the pattern after it is used to exclude tables from being processed. This effectively turns the filter into a block list. + +``` +*.* +#^ note: must add the *.* to include all tables first +!*.Password +!employees.salaries +``` + +### Escape character + +To turn a special character into an identifier character, precede it with a backslash `\`. + +``` +db\.with\.dots.* +``` + +For simplicity and future compatibility, the following sequences are prohibited: + +* `\` at the end of the line after trimming whitespaces (use `[ ]` to match a literal whitespace at the end). +* `\` followed by any ASCII alphanumeric character (`[0-9a-zA-Z]`). In particular, C-like escape sequences like `\0`, `\r`, `\n` and `\t` currently are meaningless. + +### Quoted identifier + +Besides `\`, special characters can also be suppressed by quoting using `"` or `` ` ``. + +``` +"db.with.dots"."tbl\1" +`db.with.dots`.`tbl\2` +``` + +The quotation mark can be included within an identifier by doubling itself. + +``` +"foo""bar".`foo``bar` +# equivalent to: +foo\"bar.foo\`bar +``` + +Quoted identifiers cannot span multiple lines. + +It is invalid to partially quote an identifier: + +``` +"this is "invalid*.* +``` + +### Regular expression + +In case very complex rules are needed, each pattern can be written as a regular expression delimited with `/`: + +``` +/^db\d{2,}$/./^tbl\d{2,}$/ +``` + +These regular expressions use the [Go dialect](https://pkg.go.dev/regexp/syntax?tab=doc). The pattern is matched if the identifier contains a substring matching the regular expression. For instance, `/b/` matches `db01`. + +> **Note:** +> +> Every `/` in the regular expression must be escaped as `\/`, including inside `[…]`. You cannot place an unescaped `/` between `\Q…\E`. + +## Multiple rules + +When a table name matches none of the rules in the filter list, the default behavior is to ignore such unmatched tables. + +To build a block list, an explicit `*.*` must be used as the first rule, otherwise all tables will be excluded. + +```bash +# every table will be filtered out +./dumpling -f '!*.Password' + +# only the "Password" table is filtered out, the rest are included. +./dumpling -f '*.*' -f '!*.Password' +``` + +In a filter list, if a table name matches multiple patterns, the last match decides the outcome. For instance: + +``` +# rule 1 +employees.* +# rule 2 +!*.dep* +# rule 3 +*.departments +``` + +The filtered outcome is as follows: + +| Table name | Rule 1 | Rule 2 | Rule 3 | Outcome | +|-----------------------|--------|--------|--------|------------------| +| irrelevant.table | | | | Default (reject) | +| employees.employees | ✓ | | | Rule 1 (accept) | +| employees.dept_emp | ✓ | ✓ | | Rule 2 (reject) | +| employees.departments | ✓ | ✓ | ✓ | Rule 3 (accept) | +| else.departments | | ✓ | ✓ | Rule 3 (accept) | + +> **Note:** +> +> In TiDB tools, the system schemas are always excluded regardless of the table filter settings. The system schemas are: +> +> * `INFORMATION_SCHEMA` +> * `PERFORMANCE_SCHEMA` +> * `METRICS_SCHEMA` +> * `INSPECTION_SCHEMA` +> * `mysql` +> * `sys` diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index 6ea6dc77f96b4..daa1413dd2ed4 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -134,6 +134,26 @@ no-schema = false # schema encoding. character-set = "auto" +<<<<<<< HEAD +======= +# Assumes the input data are "strict" to speed up processing. +# Implications of strict-format = true are: +# * in CSV, every value cannot contain literal new lines (U+000A and U+000D, or \r and \n) even +# when quoted, i.e. new lines are strictly used to separate rows. +# Strict format allows Lightning to quickly locate split positions of a large file for parallel +# processing. However, if the input data is not strict, it may split a valid data in half and +# corrupt the result. +# The default value is false for safety over speed. +strict-format = false + +# If strict-format is true, Lightning will split large CSV files into multiple chunks to process in +# parallel. max-region-size is the maximum size of each chunk after splitting. +# max-region-size = 268_435_456 # Byte (default = 256 MB) + +# Only import tables if these wildcard rules are matched. See the corresponding section for details. +filter = ['*.*'] + +>>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) # Configures how CSV files are parsed. [mydumper.csv] # Separator between fields, should be an ASCII character. @@ -205,10 +225,13 @@ analyze = true switch-mode = "5m" # Duration between which an import progress is printed to the log. log-progress = "5m" +<<<<<<< HEAD # Table filter options. See the corresponding section for details. # [black-white-list] # ... +======= +>>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) ``` ### TiKV Importer @@ -289,7 +312,13 @@ min-available-ratio = 0.05 | -V | Prints program version | | | -d *directory* | Directory of the data dump to read from | `mydumper.data-source-dir` | | -L *level* | Log level: debug, info, warn, error, fatal (default = info) | `lightning.log-level` | +<<<<<<< HEAD | --log-file *file* | Log file path | `lightning.log-file` | +======= +| -f *rule* | [Table filter rules](/table-filter.md) (can be specified multiple times) | `mydumper.filter` | +| --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-tidb-backend.md) (`importer` or `tidb`) | `tikv-importer.backend` | +| --log-file *file* | Log file path (default = a temporary file in `/tmp`) | `lightning.log-file` | +>>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) | --status-addr *ip:port* | Listening address of the TiDB Lightning server | `lightning.status-port` | | --importer *host:port* | Address of TiKV Importer | `tikv-importer.addr` | | --pd-urls *host:port* | PD endpoint address | `tidb.pd-addr` | diff --git a/tidb-lightning/tidb-lightning-glossary.md b/tidb-lightning/tidb-lightning-glossary.md index cf9edf8d07573..3774aa2a7a0c8 100644 --- a/tidb-lightning/tidb-lightning-glossary.md +++ b/tidb-lightning/tidb-lightning-glossary.md @@ -35,12 +35,6 @@ Back end is the destination where TiDB Lightning sends the parsed result. Also s See [TiDB Lightning TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) for details. -### Black-white list - -A configuration list that specifies which tables to be imported and which should be excluded. - -See [TiDB Lightning Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) for details. - ## C @@ -101,6 +95,16 @@ Engines use TiKV Importer's `import-dir` as temporary storage, which are sometim See also [data engine](/tidb-lightning/tidb-lightning-glossary.md#data-engine) and [index engine](/tidb-lightning/tidb-lightning-glossary.md#index-engine). + + +## F + +### Filter + +A configuration list that specifies which tables to be imported or excluded. + +See [Table Filter](/table-filter.md) for details. + ## I From daa9f6070f685a2cd5d3e02957d70c9b84f124dd Mon Sep 17 00:00:00 2001 From: Ran Date: Fri, 3 Jul 2020 16:15:52 +0800 Subject: [PATCH 2/5] resolve conflicts --- TOC.md | 237 +-------------------------------------------------------- 1 file changed, 1 insertion(+), 236 deletions(-) diff --git a/TOC.md b/TOC.md index 4bf2e31090dca..dd3f3d2529816 100644 --- a/TOC.md +++ b/TOC.md @@ -92,7 +92,6 @@ - [Identify Slow Queries](/identify-slow-queries.md) - [Identify Expensive Queries](/identify-expensive-queries.md) + Scale -<<<<<<< HEAD - [Scale using Ansible](/scale-tidb-using-ansible.md) + Upgrade - [Upgrade to TiDB 3.1](/upgrade-tidb-using-ansible.md) @@ -100,130 +99,6 @@ - [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - [Troubleshoot TiDB Lightning](/troubleshoot-tidb-lightning.md) -======= - + [Use TiUP (Recommended)](/scale-tidb-using-tiup.md) - + [Use TiDB Ansible](/scale-tidb-using-ansible.md) - + [Use TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/scale-a-tidb-cluster) - + Backup and Restore - + [Use Mydumper and TiDB Lightning](/backup-and-restore-using-mydumper-lightning.md) - + [Use Dumpling for Export or Backup](/export-or-backup-using-dumpling.md) - + Use BR Tool - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + [BR storages](/br/backup-and-restore-storages.md) - + [Configure Time Zone](/configure-time-zone.md) - + [Daily Checklist](/daily-check.md) - + [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) - + [Maintain TiFlash](/tiflash/maintain-tiflash.md) - + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) - + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) -+ Monitor and Alert - + [Monitoring Framework Overview](/tidb-monitoring-framework.md) - + [Monitoring API](/tidb-monitoring-api.md) - + [Deploy Monitoring Services](/deploy-monitoring-services.md) - + [TiDB Cluster Alert Rules](/alert-rules.md) - + [TiFlash Alert Rules](/tiflash/tiflash-alert-rules.md) -+ Troubleshoot - + [Identify Slow Queries](/identify-slow-queries.md) - + [SQL Diagnostics](/system-tables/system-table-sql-diagnostics.md) - + [Identify Expensive Queries](/identify-expensive-queries.md) - + [Statement Summary Tables](/statement-summary-tables.md) - + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) - + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) - + [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) - + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) -+ Performance Tuning - + System Tuning - + [Operating System Tuning](/tune-operating-system.md) - + Software Tuning - + Configuration - + [Tune TiDB Memory](/configure-memory-usage.md) - + [Tune TiKV Threads](/tune-tikv-thread-performance.md) - + [Tune TiKV Memory](/tune-tikv-memory-performance.md) - + [TiKV Follower Read](/follower-read.md) - + [TiFlash Tuning](/tiflash/tune-tiflash-performance.md) - + [Coprocessor Cache](/coprocessor-cache.md) - + SQL Tuning - + [SQL Tuning with `EXPLAIN`](/query-execution-plan.md) - + SQL Optimization - + [SQL Optimization Process](/sql-optimization-concepts.md) - + Logic Optimization - + [Join Reorder](/join-reorder.md) - + Physical Optimization - + [Statistics](/statistics.md) - + Control Execution Plan - + [Optimizer Hints](/optimizer-hints.md) - + [SQL Plan Management](/sql-plan-management.md) - + [Access Tables Using `IndexMerge`](/index-merge.md) -+ Tutorials - + [Multiple Data Centers in One City Deployment](/multi-data-centers-in-one-city-deployment.md) - + [Three Data Centers in Two Cities Deployment](/three-data-centers-in-two-cities-deployment.md) - + Best Practices - + [Use TiDB](/tidb-best-practices.md) - + [Java Application Development](/best-practices/java-app-best-practices.md) - + [Use HAProxy](/best-practices/haproxy-best-practices.md) - + [Highly Concurrent Write](/best-practices/high-concurrency-best-practices.md) - + [Grafana Monitoring](/best-practices/grafana-monitor-best-practices.md) - + [PD Scheduling](/best-practices/pd-scheduling-best-practices.md) - + [TiKV Performance Tuning with Massive Regions](/best-practices/massive-regions-best-practices.md) - + [Use Placement Rules](/configure-placement-rules.md) - + [Use Load Base Split](/configure-load-base-split.md) - + [Use Store Limit](/configure-store-limit.md) -+ TiDB Ecosystem Tools - + [Overview](/ecosystem-tool-user-guide.md) - + [Use Cases](/ecosystem-tool-user-case.md) - + [Download](/download-ecosystem-tools.md) - + Backup & Restore (BR) - + [BR FAQ](/br/backup-and-restore-faq.md) - + [Use BR Tool](/br/backup-and-restore-tool.md) - + [BR Use Cases](/br/backup-and-restore-use-cases.md) - + TiDB Binlog - + [Overview](/tidb-binlog/tidb-binlog-overview.md) - + [Deploy](/tidb-binlog/deploy-tidb-binlog.md) - + [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md) - + [Configure](/tidb-binlog/tidb-binlog-configuration-file.md) - + [Pump](/tidb-binlog/tidb-binlog-configuration-file.md#pump) - + [Drainer](/tidb-binlog/tidb-binlog-configuration-file.md#drainer) - + [Upgrade](/tidb-binlog/upgrade-tidb-binlog.md) - + [Monitor](/tidb-binlog/monitor-tidb-binlog-cluster.md) - + [Reparo](/tidb-binlog/tidb-binlog-reparo.md) - + [binlogctl](/tidb-binlog/binlog-control.md) - + [Binlog Slave Client](/tidb-binlog/binlog-slave-client.md) - + [TiDB Binlog Relay Log](/tidb-binlog/tidb-binlog-relay-log.md) - + [Bidirectional Replication Between TiDB Clusters](/tidb-binlog/bidirectional-replication-between-tidb-clusters.md) - + [Glossary](/tidb-binlog/tidb-binlog-glossary.md) - + Troubleshoot - + [Troubleshoot](/tidb-binlog/troubleshoot-tidb-binlog.md) - + [Handle Errors](/tidb-binlog/handle-tidb-binlog-errors.md) - + [FAQ](/tidb-binlog/tidb-binlog-faq.md) - + TiDB Lightning - + [Overview](/tidb-lightning/tidb-lightning-overview.md) - + [Tutorial](/get-started-with-tidb-lightning.md) - + [Deploy](/tidb-lightning/deploy-tidb-lightning.md) - + [Configure](/tidb-lightning/tidb-lightning-configuration.md) - + Key Features - + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) - + [Table Filter](/table-filter.md) - + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) - + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) - + [Monitor](/tidb-lightning/monitor-tidb-lightning.md) - + [Troubleshoot](/troubleshoot-tidb-lightning.md) - + [FAQ](/tidb-lightning/tidb-lightning-faq.md) - + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) - + [TiCDC](/ticdc/ticdc-overview.md) - + sync-diff-inspector - + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) - + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) - + [Data Check in Sharding Scenarios](/sync-diff-inspector/shard-diff.md) - + [Data Check for TiDB Upstream/Downstream Clusters](/sync-diff-inspector/upstream-downstream-diff.md) - + [Loader](/loader-overview.md) - + [Mydumper](/mydumper-overview.md) - + [Syncer](/syncer-overview.md) - + TiSpark - + [Quick Start](/get-started-with-tispark.md) - + [User Guide](/tispark-overview.md) ->>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) + Reference + SQL - [MySQL Compatibility](/mysql-compatibility.md) @@ -273,7 +148,6 @@ - [`SET`](/data-type-string.md#set-type) - [JSON Type](/data-type-json.md) + Functions and Operators -<<<<<<< HEAD - [Function and Operator Reference](/functions-and-operators/functions-and-operators-overview.md) - [Type Conversion in Expression Evaluation](/functions-and-operators/type-conversion-in-expression-evaluation.md) - [Operators](/functions-and-operators/operators.md) @@ -482,7 +356,7 @@ - [Deployment](/tidb-lightning/deploy-tidb-lightning.md) - [Configuration](/tidb-lightning/tidb-lightning-configuration.md) - [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) - - [Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) + - [Table Filter](/table-filter.md) - [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) - [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) - [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) @@ -500,115 +374,6 @@ - [TiKV Control](/tikv-control.md) - [TiDB Control](/tidb-control.md) - [TiDB in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/) -======= - + [Overview](/functions-and-operators/functions-and-operators-overview.md) - + [Type Conversion in Expression Evaluation](/functions-and-operators/type-conversion-in-expression-evaluation.md) - + [Operators](/functions-and-operators/operators.md) - + [Control Flow Functions](/functions-and-operators/control-flow-functions.md) - + [String Functions](/functions-and-operators/string-functions.md) - + [Numeric Functions and Operators](/functions-and-operators/numeric-functions-and-operators.md) - + [Date and Time Functions](/functions-and-operators/date-and-time-functions.md) - + [Bit Functions and Operators](/functions-and-operators/bit-functions-and-operators.md) - + [Cast Functions and Operators](/functions-and-operators/cast-functions-and-operators.md) - + [Encryption and Compression Functions](/functions-and-operators/encryption-and-compression-functions.md) - + [Information Functions](/functions-and-operators/information-functions.md) - + [JSON Functions](/functions-and-operators/json-functions.md) - + [Aggregate (GROUP BY) Functions](/functions-and-operators/aggregate-group-by-functions.md) - + [Window Functions](/functions-and-operators/window-functions.md) - + [Miscellaneous Functions](/functions-and-operators/miscellaneous-functions.md) - + [Precision Math](/functions-and-operators/precision-math.md) - + [List of Expressions for Pushdown](/functions-and-operators/expressions-pushed-down.md) - + [Constraints](/constraints.md) - + [Generated Columns](/generated-columns.md) - + [SQL Mode](/sql-mode.md) - + Transactions - + [Overview](/transaction-overview.md) - + [Isolation Levels](/transaction-isolation-levels.md) - + [Optimistic Transactions](/optimistic-transaction.md) - + [Pessimistic Transactions](/pessimistic-transaction.md) - + Garbage Collection (GC) - + [Overview](/garbage-collection-overview.md) - + [Configuration](/garbage-collection-configuration.md) - + [Views](/views.md) - + [Partitioning](/partitioned-table.md) - + [Character Set and Collation](/character-set-and-collation.md) - + System Tables - + [`mysql`](/system-tables/system-table-overview.md) - + [`information_schema`](/system-tables/system-table-information-schema.md) - + sql-diagnosis - + [`cluster_info`](/system-tables/system-table-cluster-info.md) - + [`cluster_hardware`](/system-tables/system-table-cluster-hardware.md) - + [`cluster_config`](/system-tables/system-table-cluster-config.md) - + [`cluster_load`](/system-tables/system-table-cluster-load.md) - + [`cluster_systeminfo`](/system-tables/system-table-cluster-systeminfo.md) - + [`cluster_log`](/system-tables/system-table-cluster-log.md) - + [`metrics_schema`](/system-tables/system-table-metrics-schema.md) - + [`metrics_tables`](/system-tables/system-table-metrics-tables.md) - + [`metrics_summary`](/system-tables/system-table-metrics-summary.md) - + [`inspection_result`](/system-tables/system-table-inspection-result.md) - + [`inspection_summary`](/system-tables/system-table-inspection-summary.md) - + UI - + TiDB Dashboard - + [Overview](/dashboard/dashboard-intro.md) - + Maintain - + [Deploy](/dashboard/dashboard-ops-deploy.md) - + [Reverse Proxy](/dashboard/dashboard-ops-reverse-proxy.md) - + [Secure](/dashboard/dashboard-ops-security.md) - + [Access](/dashboard/dashboard-access.md) - + [Overview Page](/dashboard/dashboard-overview.md) - + [Cluster Info Page](/dashboard/dashboard-cluster-info.md) - + [Key Visualizer Page](/dashboard/dashboard-key-visualizer.md) - + SQL Statements Analysis - + [SQL Statements Page](/dashboard/dashboard-statement-list.md) - + [SQL Details Page](/dashboard/dashboard-statement-details.md) - + [Slow Queries Page](/dashboard/dashboard-slow-query.md) - + Cluster Diagnostics - + [Access Cluster Diagnostics Page](/dashboard/dashboard-diagnostics-access.md) - + [View Diagnostics Report](/dashboard/dashboard-diagnostics-report.md) - + [Use Diagnostics](/dashboard/dashboard-diagnostics-usage.md) - + [Search Logs Page](/dashboard/dashboard-log-search.md) - + [Profile Instances Page](/dashboard/dashboard-profiling.md) - + [FAQ](/dashboard/dashboard-faq.md) - + CLI - + [tikv-ctl](/tikv-control.md) - + [pd-ctl](/pd-control.md) - + [tidb-ctl](/tidb-control.md) - + [pd-recover](/pd-recover.md) - + Command Line Flags - + [tidb-server](/command-line-flags-for-tidb-configuration.md) - + [tikv-server](/command-line-flags-for-tikv-configuration.md) - + [tiflash-server](/tiflash/tiflash-command-line-flags.md) - + [pd-server](/command-line-flags-for-pd-configuration.md) - + Configuration File Parameters - + [tidb-server](/tidb-configuration-file.md) - + [tikv-server](/tikv-configuration-file.md) - + [tiflash-server](/tiflash/tiflash-configuration.md) - + [pd-server](/pd-configuration-file.md) - + System Variables - + [MySQL System Variables](/system-variables.md) - + [TiDB Specific System Variables](/tidb-specific-system-variables.md) - + Storage Engines - + TiFlash - + [Overview](/tiflash/tiflash-overview.md) - + [Use TiFlash](/tiflash/use-tiflash.md) - + TiUP - + [Documentation Guide](/tiup/tiup-documentation-guide.md) - + [Overview](/tiup/tiup-overview.md) - + [Terminology and Concepts](/tiup/tiup-terminology-and-concepts.md) - + [Manage TiUP Components](/tiup/tiup-component-management.md) - + [FAQ](/tiup/tiup-faq.md) - + [Troubleshooting Guide](/tiup/tiup-troubleshooting-guide.md) - + TiUP Components - + [tiup-playground](/tiup/tiup-playground.md) - + [tiup-cluster](/tiup/tiup-cluster.md) - + [tiup-mirror](/tiup/tiup-mirror.md) - + [tiup-bench](/tiup/tiup-bench.md) - + [Telemetry](/telemetry.md) - + [Errors Codes](/error-codes.md) - + [TiCDC Overview](/ticdc/ticdc-overview.md) - + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) - + [Table Filter](/table-filter.md) ->>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) + FAQs - [TiDB FAQs](/faq/tidb-faq.md) - [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md) From 43c6e18fee5e01907d31a9f4bba6597e08a151e0 Mon Sep 17 00:00:00 2001 From: Ran Date: Fri, 3 Jul 2020 16:29:07 +0800 Subject: [PATCH 3/5] resolve conflicts and update TOC --- TOC.md | 1 + table-filter.md | 2 +- .../tidb-lightning-configuration.md | 31 +------------------ 3 files changed, 3 insertions(+), 31 deletions(-) diff --git a/TOC.md b/TOC.md index dd3f3d2529816..679be45e9342a 100644 --- a/TOC.md +++ b/TOC.md @@ -343,6 +343,7 @@ - [Overview](/ecosystem-tool-user-guide.md) - [Use Cases](/ecosystem-tool-user-case.md) - [Download](/download-ecosystem-tools.md) + - [Table Filter](/table-filter.md) + Backup & Restore (BR) - [Use BR](/br/backup-and-restore-tool.md) - [BR Use Cases](/br/backup-and-restore-use-cases.md) diff --git a/table-filter.md b/table-filter.md index 80f44ac522ec8..edb2c429bc07f 100644 --- a/table-filter.md +++ b/table-filter.md @@ -2,7 +2,7 @@ title: Table Filter summary: Usage of table filter feature in TiDB tools. category: reference -aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/','/docs/dev/reference/tools/tidb-lightning/table-filter/','/tidb/dev/tidb-lightning-table-filter/'] +aliases: ['/docs/v3.1/tidb-lightning/tidb-lightning-table-filter/','/docs/v3.1/reference/tools/tidb-lightning/table-filter/','/tidb/v3.1/tidb-lightning-table-filter/'] --- # Table Filter diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index daa1413dd2ed4..cc40577765d6d 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -134,26 +134,9 @@ no-schema = false # schema encoding. character-set = "auto" -<<<<<<< HEAD -======= -# Assumes the input data are "strict" to speed up processing. -# Implications of strict-format = true are: -# * in CSV, every value cannot contain literal new lines (U+000A and U+000D, or \r and \n) even -# when quoted, i.e. new lines are strictly used to separate rows. -# Strict format allows Lightning to quickly locate split positions of a large file for parallel -# processing. However, if the input data is not strict, it may split a valid data in half and -# corrupt the result. -# The default value is false for safety over speed. -strict-format = false - -# If strict-format is true, Lightning will split large CSV files into multiple chunks to process in -# parallel. max-region-size is the maximum size of each chunk after splitting. -# max-region-size = 268_435_456 # Byte (default = 256 MB) - # Only import tables if these wildcard rules are matched. See the corresponding section for details. filter = ['*.*'] ->>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) # Configures how CSV files are parsed. [mydumper.csv] # Separator between fields, should be an ASCII character. @@ -225,13 +208,6 @@ analyze = true switch-mode = "5m" # Duration between which an import progress is printed to the log. log-progress = "5m" -<<<<<<< HEAD - -# Table filter options. See the corresponding section for details. -# [black-white-list] -# ... -======= ->>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) ``` ### TiKV Importer @@ -312,13 +288,8 @@ min-available-ratio = 0.05 | -V | Prints program version | | | -d *directory* | Directory of the data dump to read from | `mydumper.data-source-dir` | | -L *level* | Log level: debug, info, warn, error, fatal (default = info) | `lightning.log-level` | -<<<<<<< HEAD -| --log-file *file* | Log file path | `lightning.log-file` | -======= | -f *rule* | [Table filter rules](/table-filter.md) (can be specified multiple times) | `mydumper.filter` | -| --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-tidb-backend.md) (`importer` or `tidb`) | `tikv-importer.backend` | -| --log-file *file* | Log file path (default = a temporary file in `/tmp`) | `lightning.log-file` | ->>>>>>> 68375a7... tidb-lightning,br: replaced black-white-list by table-filter (#3065) +| --log-file *file* | Log file path | `lightning.log-file` | | --status-addr *ip:port* | Listening address of the TiDB Lightning server | `lightning.status-port` | | --importer *host:port* | Address of TiKV Importer | `tikv-importer.addr` | | --pd-urls *host:port* | PD endpoint address | `tidb.pd-addr` | From 19573e62b6ad0d805deb0f4f018875f3bec825e4 Mon Sep 17 00:00:00 2001 From: Ran Date: Fri, 3 Jul 2020 16:37:29 +0800 Subject: [PATCH 4/5] remove ticdc which is for 4.0 --- table-filter.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/table-filter.md b/table-filter.md index edb2c429bc07f..7508d96a11b00 100644 --- a/table-filter.md +++ b/table-filter.md @@ -57,17 +57,6 @@ Table filters in TOML files are specified as [array of strings](https://toml.io/ filter = ['foo*.*', 'bar*.*'] ``` -* [TiCDC](/ticdc/ticdc-overview.md): - - ```toml - [filter] - rules = ['foo*.*', 'bar*.*'] - - [[sink.dispatchers]] - matcher = ['db1.*', 'db2.*', 'db3.*'] - dispatcher = 'ts' - ``` - ## Syntax ### Plain table names From 8c5533d6e9c19171bd64f8cd308683106c28345c Mon Sep 17 00:00:00 2001 From: Ran Date: Mon, 6 Jul 2020 14:12:16 +0800 Subject: [PATCH 5/5] Update table-filter.md --- table-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/table-filter.md b/table-filter.md index 7508d96a11b00..75e1eed3c21bb 100644 --- a/table-filter.md +++ b/table-filter.md @@ -9,7 +9,7 @@ aliases: ['/docs/v3.1/tidb-lightning/tidb-lightning-table-filter/','/docs/v3.1/r The TiDB ecosystem tools operate on all the databases by default, but oftentimes only a subset is needed. For example, you only want to work with the schemas in the form of `foo*` and `bar*` and nothing else. -Since TiDB 4.0, all TiDB ecosystem tools share a common filter syntax to define subsets. This document describes how to use the table filter feature. +Several TiDB ecosystem tools share a common filter syntax to define subsets. This document describes how to use the table filter feature. ## Usage