Skip to content

Commit

Permalink
Support setting timezone for stream load and routine load (#1831)
Browse files Browse the repository at this point in the history
  • Loading branch information
morningman authored and imay committed Sep 19, 2019
1 parent 7bf02d0 commit e8da855
Show file tree
Hide file tree
Showing 11 changed files with 426 additions and 231 deletions.
3 changes: 3 additions & 0 deletions be/src/http/action/stream_load.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,9 @@ Status StreamLoadAction::_process_put(HttpRequest* http_req, StreamLoadContext*
return Status::InvalidArgument("Invalid strict mode format. Must be bool type");
}
}
if (!http_req->header(HTTP_TIMEZONE).empty()) {
request.__set_timezone(http_req->header(HTTP_TIMEZONE));
}

// plan this load
TNetworkAddress master_addr = _exec_env->master_info()->network_address;
Expand Down
1 change: 1 addition & 0 deletions be/src/http/http_common.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ static const std::string HTTP_TIMEOUT = "timeout";
static const std::string HTTP_PARTITIONS = "partitions";
static const std::string HTTP_NEGATIVE = "negative";
static const std::string HTTP_STRICT_MODE = "strict_mode";
static const std::string HTTP_TIMEZONE = "timezone";

static const std::string HTTP_100_CONTINUE = "100-continue";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@

是否开启严格模式,默认为开启。如果开启后,非空原始数据的列类型变换如果结果为 NULL,则会被过滤。指定方式为 "strict_mode" = "true"

5. timezone
指定导入作业所使用的时区。默认为使用 Session 的 timezone 参数。该参数会影响所有导入涉及的和时区有关的函数结果。

5. data_source

数据源的类型。当前支持:
Expand Down Expand Up @@ -161,6 +165,7 @@

"kafka_partitions" = "0,1,2,3",
"kafka_offsets" = "101,0,OFFSET_BEGINNING,OFFSET_END"
4. property

指定自定义kafka参数。
Expand Down Expand Up @@ -232,7 +237,7 @@
"kafka_offsets" = "101,0,0,200"
);

2. 通过 SSL 认证方式,从 Kafka 集群导入数据。同时设置 client.id 参数。导入任务为非严格模式
2. 通过 SSL 认证方式,从 Kafka 集群导入数据。同时设置 client.id 参数。导入任务为非严格模式,时区为 Africa/Abidjan

CREATE ROUTINE LOAD example_db.test1 ON example_tbl
COLUMNS(k1, k2, k3, v1, v2, v3 = k1 * 100),
Expand All @@ -243,7 +248,8 @@
"max_batch_interval" = "20",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200",
"strict_mode" = "false"
"strict_mode" = "false",
"timezone" = "Africa/Abidjan"
)
FROM KAFKA
(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@
strict_mode: 用户指定此次导入是否开启严格模式,默认为开启。关闭方式为 -H "strict_mode: false"。

timezone: 指定本次导入所使用的时区。默认为东八区。该参数会影响所有导入涉及的和时区有关的函数结果。

RETURN VALUES
导入完成后,会以Json格式返回这次导入的相关内容。当前包括一下字段
Status: 导入最后的状态。
Expand Down Expand Up @@ -91,8 +93,8 @@
7. 导入含有HLL列的表,可以是表中的列或者数据中的列用于生成HLL列
curl --location-trusted -u root -H "columns: k1, k2, v1=hll_hash(k1)" -T testData http://host:port/api/testDb/testTbl/_stream_load

8. 导入数据进行严格模式过滤
curl --location-trusted -u root -H "strict_mode: true" -T testData http://host:port/api/testDb/testTbl/_stream_load
8. 导入数据进行严格模式过滤,并设置时区为 Africa/Abidjan
curl --location-trusted -u root -H "strict_mode: true" -H "timezone: Africa/Abidjan" -T testData http://host:port/api/testDb/testTbl/_stream_load

9. 导入含有聚合模型为BITMAP_UNION列的表,可以是表中的列或者数据中的列用于生成BITMAP_UNION列
curl --location-trusted -u root -H "columns: k1, k2, v1=to_bitmap(k1)" -T testData http://host:port/api/testDb/testTbl/_stream_load
Expand Down

0 comments on commit e8da855

Please sign in to comment.