From 5ec111a2e3b16e6c2b3b2c3b2746ec508e7e212e Mon Sep 17 00:00:00 2001 From: stalary Date: Sat, 20 Jan 2024 22:22:23 +0800 Subject: [PATCH 1/3] chore: fix doc MySql Load format --- .../data-operate/import/import-scenes/local-file-load.md | 8 ++++---- .../data-operate/import/import-scenes/local-file-load.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md index 187bc5e39bc597..51e8464961cd43 100644 --- a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md +++ b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md @@ -224,9 +224,9 @@ public class DorisStreamLoader { > > ``` -## MySql LOAD +## MySql Load - Example of mysql load + Example of MySql Load ### Import Data @@ -255,7 +255,7 @@ public class DorisStreamLoader { INTO TABLE demo.load_local_file_test ``` - For more advanced operations of the MySQL Load command, see [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) Command documentation. + For more advanced operations of the MySql Load command, see [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) Command documentation. 3. Wait for the import result @@ -267,7 +267,7 @@ public class DorisStreamLoader { ``` - Load success if the client show the return rows. Otherwise sql statement will throw an exception and show the error message in client. - - For details of other fields, please refer to the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command documentation. + - For details of other fields, please refer to the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command documentation. ### Import Suggestion diff --git a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md index be023fe3540a43..723a4117d3ec6f 100644 --- a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md +++ b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md @@ -226,9 +226,9 @@ public class DorisStreamLoader { > > ``` -## MySql LOAD +## MySql Load - MySql LOAD样例 + MySql Load样例 ### 导入数据 @@ -257,7 +257,7 @@ public class DorisStreamLoader { INTO TABLE demo.load_local_file_test ``` - 关于 MySQL Load 命令的更多高级操作，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 + 关于 MySql Load 命令的更多高级操作，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 3. 等待导入结果 @@ -269,7 +269,7 @@ public class DorisStreamLoader { ``` - 如果出现上述结果, 则表示导入成功。导入失败, 会抛出错误,并在客户端显示错误原因 - - 其他字段的详细介绍，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 + - 其他字段的详细介绍，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 ### 导入建议 - MySql Load 只能导入本地文件(可以是客户端本地或者连接的FE节点本地), 而且支持CSV格式。 From 15a104d7b07e25623905d89436d3abc1da1fdf27 Mon Sep 17 00:00:00 2001 From: stalary Date: Mon, 19 Feb 2024 15:53:51 +0800 Subject: [PATCH 2/3] chore: modify mysql-load-manual.md --- .../import/import-way/mysql-load-manual.md | 24 ++++++++-------- .../import/import-way/mysql-load-manual.md | 28 +++++++++---------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md index 06592878c8954a..78130079bb0074 100644 --- a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md +++ b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md @@ -24,7 +24,7 @@ specific language governing permissions and limitations under the License. --> -# Mysql Load +# MySql Load This is an stand syntax of MySql [LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html) for user to load local file. @@ -37,9 +37,9 @@ MySql load is mainly suitable for importing local files on the client side, or i ## Basic Principles -The MySql Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySQL Load will reuses StreamLoad: +The MySql Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySql Load will reuses StreamLoad: - 1. FE receives the MySQL Load request executed by the client and then analyse the SQL + 1. FE receives the MySql Load request executed by the client and then analyse the SQL 2. FE build the MySql Load request as a StreamLoad request. @@ -61,7 +61,7 @@ MySql Load currently only supports data formats: CSV (text). mysql --local-infile -h 127.0.0.1 -P 9030 -u root -D testdb ``` -Notice that if you wants to use mysql load, you must connect doris server with `--local-infile` in client command. +Notice that if you wants to use MySql load, you must connect doris server with `--local-infile` in client command. If you're use jdbc to connect doris, you must add property named `allowLoadLocalInfile=true` in jdbc url. @@ -70,7 +70,7 @@ If you're use jdbc to connect doris, you must add property named `allowLoadLocal CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY hash (pk) PROPERTIES ('replication_num' = '1'); ``` ### import file from client node - Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySQL LOAD syntax. + Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySql Load syntax. ```sql LOAD DATA LOCAL @@ -84,7 +84,7 @@ IGNORE 1 LINES SET (c1 = k1, c2 = k2, c3 = v10, c4 = v11) PROPERTIES ("strict_mode" = "true") ``` -1. MySQL Load starts with the syntax `LOAD DATA`, and specifying `LOCAL` means reading client side files. +1. MySql Load starts with the syntax `LOAD DATA`, and specifying `LOCAL` means reading client side files. 2. The local fill path will be filled after `INFILE`, which can be a relative path or an absolute path. Currently only a single file is supported, and multiple files are not supported 3. The table name after `INTO TABLE` can specify the database name, as shown in the case. It can also be omitted, and the current database for the user will be used. 4. `PARTITION` syntax supports specified partition to import @@ -92,7 +92,7 @@ PROPERTIES ("strict_mode" = "true") 6. `LINES TERMINATED BY` specifies the line separator 7. `IGNORE num LINES` skips the num header of the CSV. 8. Column mapping syntax, see the column mapping chapter of [Imported Data Transformation](../import-scenes/load-data-convert.md) for specific parameters -9. `PROPERTIES` is the configuration of import, please refer to the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual for specific properties. +9. `PROPERTIES` is the configuration of import, please refer to the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual for specific properties. ### import file from fe server node Assuming that the '/root/server_local.csv' path on the FE node is a CSV file, use the MySQL client side to connect to the corresponding FE node, and then execute the following command to import data into the test table. @@ -114,7 +114,7 @@ PROPERTIES ("strict_mode" = "true") 3. Server side load was disabled by default. Enable it by setting `mysql_load_server_secure_path` with a secure path. All the load file should be under this path. ### Return result -Since MySQL load is a synchronous import method, the imported results are returned to the user through SQL syntax. +Since MySql Load is a synchronous import method, the imported results are returned to the user through SQL syntax. If the import fails, a specific error message will be displayed. If the import is successful, the number of imported rows will be displayed. ```Text @@ -123,7 +123,7 @@ Records: 1 Deleted: 0 Skipped: 0 Warnings: 0 ``` ### Error result -If mysql load process goes wrong, it will show the error in the client as below: +If MySql Load process goes wrong, it will show the error in the client as below: ```text ERROR 1105 (HY000): errCode = 2, detailMessage = [DATA_QUALITY_ERROR]too many filtered rows with load id b612907c-ccf4-4ac2-82fe-107ece655f0f ``` @@ -139,14 +139,14 @@ The loadId was the label in this case. ### Configuration 1. `mysql_load_thread_pool`: the thread pool size for singe FE node, set 4 thread by default. The block queue size is 5 times of `mysql_load_thread_pool`. So FE can accept 4 + 4\*5 = 24 requests in one time. Increase this configuration if the parallelism are larger than 24. 2. `mysql_load_server_secure_path`: the secure path for load data from server. Empty path by default means that it's not allowed for server load. Recommend to create a `local_import_data` directory under `DORIS_HOME` to load data if you want enable it. -3. `mysql_load_in_memory_record` The failed mysql load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day. +3. `mysql_load_in_memory_record` The failed MySql Load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day. ## Notice 1. If you see this `LOAD DATA LOCAL INFILE file request rejected due to restrictions on access` message, you should connet mysql with `mysql --local-infile=1` command to enable client to load local file. -2. The configuration for StreamLoad will also affect MySQL Load. Such as the configurate in be named `streaming_load_max_mb`, it's 10GB by default and it will control the max size for one load. +2. The configuration for StreamLoad will also affect MySql Load. Such as the configurate in be named `streaming_load_max_mb`, it's 10GB by default and it will control the max size for one load. ## More Help -1. For more detailed syntax and best practices for using MySQL Load, see the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual. +1. For more detailed syntax and best practices for using MySql Load, see the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual. diff --git a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md index 7b6a66c90cbf17..e6248fb770928c 100644 --- a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md +++ b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md @@ -24,22 +24,22 @@ specific language governing permissions and limitations under the License. --> -# MySql load +# MySql Load 该语句兼容MySQL标准的[LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html)语法，方便用户导入本地数据，并降低学习成本。 -MySql load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。 +MySql Load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。 -MySql load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。 +MySql Load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。 ## 基本原理 -MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySQL Load实现复用了StreamLoad的基础导入能力: +MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySql Load实现复用了StreamLoad的基础导入能力: -1. FE接收到客户端执行的MySQL Load请求, 完成SQL解析工作 +1. FE接收到客户端执行的MySql Load请求, 完成SQL解析工作 2. FE将Load请求拆解,并封装为StreamLoad的请求. @@ -52,7 +52,7 @@ MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, ## 支持数据格式 -MySQL Load 支持数据格式：CSV（文本）。 +MySql Load 支持数据格式：CSV（文本）。 ## 基本操作举例 @@ -61,7 +61,7 @@ MySQL Load 支持数据格式：CSV（文本）。 mysql --local-infile -h 127.0.0.1 -P 9030 -u root -D testdb ``` -注意: 执行MySQL Load语句的时候, 客户端命令必须带有`--local-infile`, 否则执行可能会出现错误. 如果是通过JDBC方式连接的话, 需要在URL中需要加入配置`allowLoadLocalInfile=true` +注意: 执行MySql Load语句的时候, 客户端命令必须带有`--local-infile`, 否则执行可能会出现错误. 如果是通过JDBC方式连接的话, 需要在URL中需要加入配置`allowLoadLocalInfile=true` ### 创建测试表 @@ -70,7 +70,7 @@ CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY ha ``` ### 导入客户端文件 -假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySQL LOAD语法将表导入到测试表`testdb.t1`中. +假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySql Load语法将表导入到测试表`testdb.t1`中. ```sql LOAD DATA LOCAL @@ -84,7 +84,7 @@ IGNORE 1 LINES set (c1=k1,c2=k2,c3=v10,c4=v11) PROPERTIES ("strict_mode"="true") ``` -1. MySQL Load以语法`LOAD DATA`开头, 指定`LOCAL`表示读取客户端文件. +1. MySql Load以语法`LOAD DATA`开头, 指定`LOCAL`表示读取客户端文件. 2. `INFILE`内填写本地文件路径, 可以是相对路径, 也可以是绝对路径.目前只支持单个文件, 不支持多个文件 3. `INTO TABLE`的表名可以指定数据库名, 如案例所示. 也可以省略, 则会使用当前用户所在的数据库. 4. `PARTITION`语法支持指定分区导入 @@ -92,7 +92,7 @@ PROPERTIES ("strict_mode"="true") 6. `LINES TERMINATED BY`指定行分隔符 7. `IGNORE num LINES`用户跳过CSV的num表头. 8. 列映射语法, 具体参数详见[导入的数据转换](../import-scenes/load-data-convert.md) 的列映射章节 -9. `PROPERTIES`导入参数, 具体参数详见[MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 +9. `PROPERTIES`导入参数, 具体参数详见[MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 ### 导入服务端文件 假设在FE节点上的`/root/server_local.csv`路径为一个CSV文件, 使用MySQL客户端连接对应的FE节点, 然后执行一下命令将数据导入到测试表中. @@ -115,7 +115,7 @@ PROPERTIES ("strict_mode"="true") ### 返回结果 -由于 MySQL load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。 +由于 MySql Load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。 如果导入执行失败, 会展示具体的报错信息. 如果导入成功, 则会显示导入的行数. ```text @@ -138,7 +138,7 @@ show load warnings where label='b612907c-ccf4-4ac2-82fe-107ece655f0f'; ### 配置项 -1. `mysql_load_thread_pool`控制单个FE中MySQL Load并发执行线程个数, 默认为4. 线程池的排队队列大小为`mysql_load_thread_pool`的5倍, 因此默认情况下, 可以并发提交的任务为 4 + 4\*5 = 24个. 如果并发个数超过24时, 可以调大该配置项. +1. `mysql_load_thread_pool`控制单个FE中MySql Load并发执行线程个数, 默认为4. 线程池的排队队列大小为`mysql_load_thread_pool`的5倍, 因此默认情况下, 可以并发提交的任务为 4 + 4\*5 = 24个. 如果并发个数超过24时, 可以调大该配置项. 2. `mysql_load_server_secure_path`服务端导入的安全路径, 默认为空, 即不允许服务端导入. 如需开启这个功能, 建议在`DORIS_HOME`目录下创建一个`local_import_data`目录, 用于导入数据. 3. `mysql_load_in_memory_record`失败的任务记录个数, 该记录会保留在内存中, 默认只会保留最近的20. 如果有需要可以调大该配置. 在内存中的记录, 有效期为1天, 异步清理线程会固定一天清理一次过期数据. @@ -147,9 +147,9 @@ show load warnings where label='b612907c-ccf4-4ac2-82fe-107ece655f0f'; 1. 如果客户端出现`LOAD DATA LOCAL INFILE file request rejected due to restrictions on access`错误, 需要用`mysql --local-infile=1`命令来打开客户端的导入功能. -2. MySQL Load的导入会受到StreamLoad的配置项限制, 例如BE支持的StreamLoad最大文件量受`streaming_load_max_mb`控制, 默认为10GB. +2. MySql Load的导入会受到StreamLoad的配置项限制, 例如BE支持的StreamLoad最大文件量受`streaming_load_max_mb`控制, 默认为10GB. ## 更多帮助 -1. 关于 MySQL Load 使用的更多详细语法及最佳实践，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 +1. 关于 MySql Load 使用的更多详细语法及最佳实践，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 From 58708d54ae2178f84f9cc15bf0bef9e0940c209d Mon Sep 17 00:00:00 2001 From: stalary Date: Mon, 19 Feb 2024 16:00:36 +0800 Subject: [PATCH 3/3] chore: modify to MySQL --- .../import/import-scenes/local-file-load.md | 16 ++++---- .../import/import-way/mysql-load-manual.md | 38 +++++++++---------- .../import/import-scenes/local-file-load.md | 16 ++++---- .../import/import-way/mysql-load-manual.md | 30 +++++++-------- 4 files changed, 50 insertions(+), 50 deletions(-) diff --git a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md index 51e8464961cd43..d022ceded043ee 100644 --- a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md +++ b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md @@ -29,7 +29,7 @@ The following mainly introduces how to import local data in client. Now Doris support two way to load data from client local file: 1. [Stream Load](../import-way/stream-load-manual.md) -2. [MySql Load](../import-way/mysql-load-manual.md) +2. [MySQL Load](../import-way/mysql-load-manual.md) ## Stream Load @@ -224,9 +224,9 @@ public class DorisStreamLoader { > > ``` -## MySql Load +## MySQL Load - Example of MySql Load + Example of MySQL Load ### Import Data @@ -246,7 +246,7 @@ public class DorisStreamLoader { ```` 2. Import data - Excute fellowing sql statmeent in the mysql client to load client local file: + Excute fellowing sql statmeent in the MySQL client to load client local file: ```sql LOAD DATA @@ -255,11 +255,11 @@ public class DorisStreamLoader { INTO TABLE demo.load_local_file_test ``` - For more advanced operations of the MySql Load command, see [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) Command documentation. + For more advanced operations of the MySQL Load command, see [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) Command documentation. 3. Wait for the import result - The MySql Load command is a synchronous command, and a successful return indicates that the import is successful. If the imported data is large, a longer waiting time may be required. Examples are as follows: + The MySQL Load command is a synchronous command, and a successful return indicates that the import is successful. If the imported data is large, a longer waiting time may be required. Examples are as follows: ```text Query OK, 1 row affected (0.17 sec) @@ -267,9 +267,9 @@ public class DorisStreamLoader { ``` - Load success if the client show the return rows. Otherwise sql statement will throw an exception and show the error message in client. - - For details of other fields, please refer to the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command documentation. + - For details of other fields, please refer to the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command documentation. ### Import Suggestion - - MySql Load can only import local files(which can be client local file or fe local file) and only support csv format. + - MySQL Load can only import local files(which can be client local file or fe local file) and only support csv format. - It is recommended to limit the amount of data for an import request to 1 - 2 GB. If you have a large number of local files, you can submit them concurrently in batches. diff --git a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md index 78130079bb0074..826b03654e9324 100644 --- a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md +++ b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md @@ -1,6 +1,6 @@ --- { - "title": "MySql Load", + "title": "MySQL Load", "language": "en" } --- @@ -24,24 +24,24 @@ specific language governing permissions and limitations under the License. --> -# MySql Load +# MySQL Load -This is an stand syntax of MySql [LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html) for user to load local file. +This is an stand syntax of MySQL [LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html) for user to load local file. -MySql load synchronously executes the import and returns the import result. The return information will show whether the import is successful for user. +MySQL load synchronously executes the import and returns the import result. The return information will show whether the import is successful for user. -MySql load is mainly suitable for importing local files on the client side, or importing data from a data stream through a program. +MySQL load is mainly suitable for importing local files on the client side, or importing data from a data stream through a program. ## Basic Principles -The MySql Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySql Load will reuses StreamLoad: +The MySQL Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySQL Load will reuses StreamLoad: - 1. FE receives the MySql Load request executed by the client and then analyse the SQL + 1. FE receives the MySQL Load request executed by the client and then analyse the SQL - 2. FE build the MySql Load request as a StreamLoad request. + 2. FE build the MySQL Load request as a StreamLoad request. 3. FE selects a BE node to send a StreamLoad request @@ -52,7 +52,7 @@ The MySql Load are similar with Stream Load. Both import local files into the Do ## Support data format -MySql Load currently only supports data formats: CSV (text). +MySQL Load currently only supports data formats: CSV (text). ## Basic operations @@ -61,7 +61,7 @@ MySql Load currently only supports data formats: CSV (text). mysql --local-infile -h 127.0.0.1 -P 9030 -u root -D testdb ``` -Notice that if you wants to use MySql load, you must connect doris server with `--local-infile` in client command. +Notice that if you wants to use MySQL load, you must connect doris server with `--local-infile` in client command. If you're use jdbc to connect doris, you must add property named `allowLoadLocalInfile=true` in jdbc url. @@ -70,7 +70,7 @@ If you're use jdbc to connect doris, you must add property named `allowLoadLocal CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY hash (pk) PROPERTIES ('replication_num' = '1'); ``` ### import file from client node - Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySql Load syntax. + Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySQL Load syntax. ```sql LOAD DATA LOCAL @@ -84,7 +84,7 @@ IGNORE 1 LINES SET (c1 = k1, c2 = k2, c3 = v10, c4 = v11) PROPERTIES ("strict_mode" = "true") ``` -1. MySql Load starts with the syntax `LOAD DATA`, and specifying `LOCAL` means reading client side files. +1. MySQL Load starts with the syntax `LOAD DATA`, and specifying `LOCAL` means reading client side files. 2. The local fill path will be filled after `INFILE`, which can be a relative path or an absolute path. Currently only a single file is supported, and multiple files are not supported 3. The table name after `INTO TABLE` can specify the database name, as shown in the case. It can also be omitted, and the current database for the user will be used. 4. `PARTITION` syntax supports specified partition to import @@ -92,7 +92,7 @@ PROPERTIES ("strict_mode" = "true") 6. `LINES TERMINATED BY` specifies the line separator 7. `IGNORE num LINES` skips the num header of the CSV. 8. Column mapping syntax, see the column mapping chapter of [Imported Data Transformation](../import-scenes/load-data-convert.md) for specific parameters -9. `PROPERTIES` is the configuration of import, please refer to the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual for specific properties. +9. `PROPERTIES` is the configuration of import, please refer to the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual for specific properties. ### import file from fe server node Assuming that the '/root/server_local.csv' path on the FE node is a CSV file, use the MySQL client side to connect to the corresponding FE node, and then execute the following command to import data into the test table. @@ -114,7 +114,7 @@ PROPERTIES ("strict_mode" = "true") 3. Server side load was disabled by default. Enable it by setting `mysql_load_server_secure_path` with a secure path. All the load file should be under this path. ### Return result -Since MySql Load is a synchronous import method, the imported results are returned to the user through SQL syntax. +Since MySQL Load is a synchronous import method, the imported results are returned to the user through SQL syntax. If the import fails, a specific error message will be displayed. If the import is successful, the number of imported rows will be displayed. ```Text @@ -123,7 +123,7 @@ Records: 1 Deleted: 0 Skipped: 0 Warnings: 0 ``` ### Error result -If MySql Load process goes wrong, it will show the error in the client as below: +If MySQL Load process goes wrong, it will show the error in the client as below: ```text ERROR 1105 (HY000): errCode = 2, detailMessage = [DATA_QUALITY_ERROR]too many filtered rows with load id b612907c-ccf4-4ac2-82fe-107ece655f0f ``` @@ -139,14 +139,14 @@ The loadId was the label in this case. ### Configuration 1. `mysql_load_thread_pool`: the thread pool size for singe FE node, set 4 thread by default. The block queue size is 5 times of `mysql_load_thread_pool`. So FE can accept 4 + 4\*5 = 24 requests in one time. Increase this configuration if the parallelism are larger than 24. 2. `mysql_load_server_secure_path`: the secure path for load data from server. Empty path by default means that it's not allowed for server load. Recommend to create a `local_import_data` directory under `DORIS_HOME` to load data if you want enable it. -3. `mysql_load_in_memory_record` The failed MySql Load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day. +3. `mysql_load_in_memory_record` The failed MySQL Load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day. ## Notice -1. If you see this `LOAD DATA LOCAL INFILE file request rejected due to restrictions on access` message, you should connet mysql with `mysql --local-infile=1` command to enable client to load local file. -2. The configuration for StreamLoad will also affect MySql Load. Such as the configurate in be named `streaming_load_max_mb`, it's 10GB by default and it will control the max size for one load. +1. If you see this `LOAD DATA LOCAL INFILE file request rejected due to restrictions on access` message, you should connet MySQL with `mysql --local-infile=1` command to enable client to load local file. +2. The configuration for StreamLoad will also affect MySQL Load. Such as the configurate in be named `streaming_load_max_mb`, it's 10GB by default and it will control the max size for one load. ## More Help -1. For more detailed syntax and best practices for using MySql Load, see the [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual. +1. For more detailed syntax and best practices for using MySQL Load, see the [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) command manual. diff --git a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md index 723a4117d3ec6f..ed87cbf8975614 100644 --- a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md +++ b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md @@ -30,7 +30,7 @@ under the License. 目前Doris支持两种从本地导入数据的模式: 1. [Stream Load](../import-way/stream-load-manual.md) -2. [MySql Load](../import-way/mysql-load-manual.md) +2. [MySQL Load](../import-way/mysql-load-manual.md) ## Stream Load Stream Load 用于将本地文件导入到 Doris 中。 @@ -226,9 +226,9 @@ public class DorisStreamLoader { > > ``` -## MySql Load +## MySQL Load - MySql Load样例 + MySQL Load样例 ### 导入数据 @@ -248,7 +248,7 @@ public class DorisStreamLoader { ``` 2. 导入数据 - 在MySql客户端下执行以下 SQL 命令导入本地文件： + 在MySQL客户端下执行以下 SQL 命令导入本地文件： ```sql LOAD DATA @@ -257,11 +257,11 @@ public class DorisStreamLoader { INTO TABLE demo.load_local_file_test ``` - 关于 MySql Load 命令的更多高级操作，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 + 关于 MySQL Load 命令的更多高级操作，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 3. 等待导入结果 - MySql Load 命令是同步命令，返回成功即表示导入成功。如果导入数据较大，可能需要较长的等待时间。示例如下: + MySQL Load 命令是同步命令，返回成功即表示导入成功。如果导入数据较大，可能需要较长的等待时间。示例如下: ```text Query OK, 1 row affected (0.17 sec) @@ -269,8 +269,8 @@ public class DorisStreamLoader { ``` - 如果出现上述结果, 则表示导入成功。导入失败, 会抛出错误,并在客户端显示错误原因 - - 其他字段的详细介绍，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 + - 其他字段的详细介绍，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。 ### 导入建议 -- MySql Load 只能导入本地文件(可以是客户端本地或者连接的FE节点本地), 而且支持CSV格式。 +- MySQL Load 只能导入本地文件(可以是客户端本地或者连接的FE节点本地), 而且支持CSV格式。 - 建议一个导入请求的数据量控制在 1 - 2 GB 以内。如果有大量本地文件，可以分批并发提交。 diff --git a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md index e6248fb770928c..fcc170023a6688 100644 --- a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md +++ b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md @@ -1,6 +1,6 @@ --- { - "title": "MySql Load", + "title": "MySQL Load", "language": "zh-CN" } --- @@ -24,22 +24,22 @@ specific language governing permissions and limitations under the License. --> -# MySql Load +# MySQL Load 该语句兼容MySQL标准的[LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html)语法，方便用户导入本地数据，并降低学习成本。 -MySql Load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。 +MySQL Load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。 -MySql Load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。 +MySQL Load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。 ## 基本原理 -MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySql Load实现复用了StreamLoad的基础导入能力: +MySQL Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySQL Load实现复用了StreamLoad的基础导入能力: -1. FE接收到客户端执行的MySql Load请求, 完成SQL解析工作 +1. FE接收到客户端执行的MySQL Load请求, 完成SQL解析工作 2. FE将Load请求拆解,并封装为StreamLoad的请求. @@ -52,7 +52,7 @@ MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, ## 支持数据格式 -MySql Load 支持数据格式：CSV（文本）。 +MySQL Load 支持数据格式：CSV（文本）。 ## 基本操作举例 @@ -61,7 +61,7 @@ MySql Load 支持数据格式：CSV（文本）。 mysql --local-infile -h 127.0.0.1 -P 9030 -u root -D testdb ``` -注意: 执行MySql Load语句的时候, 客户端命令必须带有`--local-infile`, 否则执行可能会出现错误. 如果是通过JDBC方式连接的话, 需要在URL中需要加入配置`allowLoadLocalInfile=true` +注意: 执行MySQL Load语句的时候, 客户端命令必须带有`--local-infile`, 否则执行可能会出现错误. 如果是通过JDBC方式连接的话, 需要在URL中需要加入配置`allowLoadLocalInfile=true` ### 创建测试表 @@ -70,7 +70,7 @@ CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY ha ``` ### 导入客户端文件 -假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySql Load语法将表导入到测试表`testdb.t1`中. +假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySQL Load语法将表导入到测试表`testdb.t1`中. ```sql LOAD DATA LOCAL @@ -84,7 +84,7 @@ IGNORE 1 LINES set (c1=k1,c2=k2,c3=v10,c4=v11) PROPERTIES ("strict_mode"="true") ``` -1. MySql Load以语法`LOAD DATA`开头, 指定`LOCAL`表示读取客户端文件. +1. MySQL Load以语法`LOAD DATA`开头, 指定`LOCAL`表示读取客户端文件. 2. `INFILE`内填写本地文件路径, 可以是相对路径, 也可以是绝对路径.目前只支持单个文件, 不支持多个文件 3. `INTO TABLE`的表名可以指定数据库名, 如案例所示. 也可以省略, 则会使用当前用户所在的数据库. 4. `PARTITION`语法支持指定分区导入 @@ -92,7 +92,7 @@ PROPERTIES ("strict_mode"="true") 6. `LINES TERMINATED BY`指定行分隔符 7. `IGNORE num LINES`用户跳过CSV的num表头. 8. 列映射语法, 具体参数详见[导入的数据转换](../import-scenes/load-data-convert.md) 的列映射章节 -9. `PROPERTIES`导入参数, 具体参数详见[MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 +9. `PROPERTIES`导入参数, 具体参数详见[MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 ### 导入服务端文件 假设在FE节点上的`/root/server_local.csv`路径为一个CSV文件, 使用MySQL客户端连接对应的FE节点, 然后执行一下命令将数据导入到测试表中. @@ -115,7 +115,7 @@ PROPERTIES ("strict_mode"="true") ### 返回结果 -由于 MySql Load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。 +由于 MySQL Load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。 如果导入执行失败, 会展示具体的报错信息. 如果导入成功, 则会显示导入的行数. ```text @@ -138,7 +138,7 @@ show load warnings where label='b612907c-ccf4-4ac2-82fe-107ece655f0f'; ### 配置项 -1. `mysql_load_thread_pool`控制单个FE中MySql Load并发执行线程个数, 默认为4. 线程池的排队队列大小为`mysql_load_thread_pool`的5倍, 因此默认情况下, 可以并发提交的任务为 4 + 4\*5 = 24个. 如果并发个数超过24时, 可以调大该配置项. +1. `mysql_load_thread_pool`控制单个FE中MySQL Load并发执行线程个数, 默认为4. 线程池的排队队列大小为`mysql_load_thread_pool`的5倍, 因此默认情况下, 可以并发提交的任务为 4 + 4\*5 = 24个. 如果并发个数超过24时, 可以调大该配置项. 2. `mysql_load_server_secure_path`服务端导入的安全路径, 默认为空, 即不允许服务端导入. 如需开启这个功能, 建议在`DORIS_HOME`目录下创建一个`local_import_data`目录, 用于导入数据. 3. `mysql_load_in_memory_record`失败的任务记录个数, 该记录会保留在内存中, 默认只会保留最近的20. 如果有需要可以调大该配置. 在内存中的记录, 有效期为1天, 异步清理线程会固定一天清理一次过期数据. @@ -147,9 +147,9 @@ show load warnings where label='b612907c-ccf4-4ac2-82fe-107ece655f0f'; 1. 如果客户端出现`LOAD DATA LOCAL INFILE file request rejected due to restrictions on access`错误, 需要用`mysql --local-infile=1`命令来打开客户端的导入功能. -2. MySql Load的导入会受到StreamLoad的配置项限制, 例如BE支持的StreamLoad最大文件量受`streaming_load_max_mb`控制, 默认为10GB. +2. MySQL Load的导入会受到StreamLoad的配置项限制, 例如BE支持的StreamLoad最大文件量受`streaming_load_max_mb`控制, 默认为10GB. ## 更多帮助 -1. 关于 MySql Load 使用的更多详细语法及最佳实践，请参阅 [MySql Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。 +1. 关于 MySQL Load 使用的更多详细语法及最佳实践，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令手册。