Skip to content

Support hive external table on Ali oss.#11489

Merged
yiguolei merged 7 commits intoapache:masterfrom
Jibing-Li:oss
Aug 4, 2022
Merged

Support hive external table on Ali oss.#11489
yiguolei merged 7 commits intoapache:masterfrom
Jibing-Li:oss

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Aug 4, 2022

Proposed changes

Provide support for hive external table on Ali oss. Create table:

CREATE TABLE `test` (   `k1` integer NOT NULL,   `k2` integer NOT NULL,   `k3` integer ) 
engine=hive properties 
("database"="default", 
"table"="test2", 
"hive.metastore.uris"="thrift://host:port"
"AWS_ACCESS_KEY" = "YOUR_ALI_ACCESS_KEY",
"AWS_SECRET_KEY" = "YOUR_ALI_SECRET_KEY",
"AWS_ENDPOINT" = "YOUR_ALI_ENDPOINT",
"AWS_REGION" = "YOUR_ALI_REGION");

select result:

MySQL [test]> select * from test;
+------+------+------+
| k1   | k2   | k3   |
+------+------+------+
|    1 |    2 |    3 |
+------+------+------+
1 row in set (0.55 sec)

Problem summary

Fe doesn't support hive external table on oss. While querying for this kind of tables, we get
Not supported storage type: oss

Checklist(Required)

  1. Does it affect the original behavior: No
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:No need
    • Yes
    • No
    • No Need
  3. Has document been added or modified: No need
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies: No
    • Yes
    • No
  5. Are there any changes that cannot be rolled back: No
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Aug 4, 2022
@Jibing-Li Jibing-Li changed the title Oss Support hive external table on Ali oss. Aug 4, 2022
@1018ji
Copy link

1018ji commented Aug 4, 2022

oss 不是支持 s3 协议,不可以直接用吗

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Aug 4, 2022

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 4, 2022

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

oss 不是支持 s3 协议,不可以直接用吗

是支持的,但fe端hadoop文件系统api不支持oss://开头的路径,需要替换成s3://

@yiguolei
Copy link
Contributor

yiguolei commented Aug 4, 2022

oss 不是支持 s3 协议,不可以直接用吗

有一部分人想用 oss:// 这样啊

@yiguolei yiguolei merged commit 591b7f3 into apache:master Aug 4, 2022
yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Aug 9, 2022
@Jibing-Li Jibing-Li deleted the oss branch September 12, 2022 14:27
Henry2SS pushed a commit to Henry2SS/incubator-doris that referenced this pull request Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/multi-catalog area/planner Issues or PRs related to the query planner dev/merge-1.1.2 reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants