feat: Iceberg/create-catalog #9017

ClSlaid · 2022-11-29T14:03:42Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

This PR implements:

the creation of iceberg catalog, from SQL.
list database under iceberg catalog. (return database struct)
list table under iceberg database. (return table struct)

Note:

moved catalog option validating from intepreter to binder
stashed some unrelated logic of reading iceberg tables.
allow_insecure will also affect the catalog!

Example

-- create on aws s3
CREATE CATALOG iceberg_ctl
  TYPE=ICEBERG
  CONNECTION=( 
    URL="s3://my_bucket/path/to/db" 
    AWS_KEY_ID="<access-key>"
    AWS_SECRET_KEY="<secret_key>"
    SESSION_TOKEN="<session_token>"
  );

SHOW DATABASES IN iceberg_ctl;
-- +--------------------------+
-- | databases_in_iceberg_ctl |
-- +--------------------------+
-- | iceberg_db               |
-- +--------------------------+
-- 1 row in set
-- Time: 0.066s

SHOW TABLES IN iceberg_ctl.iceberg_db;
-- +----------------------+
-- | tables_in_iceberg_db |
-- +----------------------+
-- | iceberg_tbl          |
-- +----------------------+
-- 1 row in set
-- Time: 0.144s

Future Possibility

The metadata deserialize is mostly done with a fork of iceberg-rs. To fully implement iceberg in Databend repository may result in large size of code and drags compile time.

It may be a better choice to have a standalone databend-iceberg crate, working as a query engine, implementing validation, reading, writing, and partitioning for iceberg. And then plug it in Databend.

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

vercel · 2022-11-29T14:03:47Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
databend	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 21, 2023 at 3:39AM (UTC)

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

- introduce `flatten` catalog - all tables now created from database Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

src/meta/app/src/schema/catalog.rs

src/query/service/src/catalogs/catalog_manager.rs

src/query/storages/iceberg/src/catalog.rs

1. since iceberg introduces not realy much dependency, no conditional compile 2. iceberg don't have tanant abstraction, use `default` of `TableMeta` and `DatabaseMeta` Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

src/query/storages/iceberg/Cargo.toml

BohuTANG · 2022-12-07T02:58:02Z

Is it possible add a test for this PR?

ClSlaid · 2022-12-07T03:04:53Z

Is it possible add a test for this PR?

Working on it, I'm implementing new SQLs for listing databases and tables under different catalogs.

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

1. no awk Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

1. sql error in sqllogictests 2. upload test data in cluster Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ClSlaid · 2023-02-20T03:48:28Z

Strange failures in sql logic tests, having been fixing it...

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

This PR addes new syntax along with its implementations, resulting imcompatibility against old versions. The test should be resumed by following PRs under this topic Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

BohuTANG

👍

src/query/storages/iceberg/Cargo.toml

ClSlaid added 6 commits November 25, 2022 21:08

feat: read table meta

909e83b

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/read-metadata

9406a17

stash commit: safe progress in iceberg

c7611ce

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

feat: create iceberg catalog

97a2d35

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

d8e3b50

refactor: move parsing location to binder

34f78d8

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

mergify bot added the pr-feature this PR introduces a new feature to the codebase label Nov 29, 2022

ClSlaid added 3 commits November 29, 2022 22:31

refactor: regenerate golden file

44cb543

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

refactor: move dependency to dev

71bd88a

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

ec5f43c

vercel bot deployed to Preview December 2, 2022 03:38 View deployment

ClSlaid added 6 commits December 5, 2022 01:13

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

da42f95

list tables: stash commit

3986f5c

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

fix: no feature in binder

437c7a5

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Merge branch 'iceberg/list-hierarchy' into iceberg/create-catalog

d8c3656

feat: refactor hierarchy

42a0eb0

- introduce `flatten` catalog - all tables now created from database Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

refactor: remove outdated comments

20f3d83

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ClSlaid marked this pull request as ready for review December 5, 2022 10:17

ClSlaid requested review from Xuanwo and PsiACE as code owners December 5, 2022 10:17

Xuanwo reviewed Dec 5, 2022

View reviewed changes

src/meta/app/src/schema/catalog.rs Show resolved Hide resolved

src/query/service/src/catalogs/catalog_manager.rs Outdated Show resolved Hide resolved

src/query/storages/iceberg/src/catalog.rs Outdated Show resolved Hide resolved

ClSlaid added 4 commits December 6, 2022 13:55

refactor: enable iceberg

0a6c564

1. since iceberg introduces not realy much dependency, no conditional compile 2. iceberg don't have tanant abstraction, use `default` of `TableMeta` and `DatabaseMeta` Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

ce45bf4

refactor: bump opendal to 0.22

550248e

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

6b1ae91

vercel bot deployed to Preview December 7, 2022 02:16 View deployment

BohuTANG reviewed Dec 7, 2022

View reviewed changes

src/query/storages/iceberg/Cargo.toml Outdated Show resolved Hide resolved

vercel bot deployed to Preview February 14, 2023 08:17 View deployment

ClSlaid added 5 commits February 14, 2023 20:48

ci: fix stateful test

19b26b6

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

ci: fix stateful test

5a6a027

1. no awk Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

ci: fix integrated tests

48fd960

1. sql error in sqllogictests 2. upload test data in cluster Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

ci: fix sqllogictests

9b8fbe8

Signed-off-by: 蔡略 <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

87c93cb

vercel bot deployed to Preview February 15, 2023 14:25 View deployment

ClSlaid added 3 commits February 16, 2023 15:28

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

d5afa3d

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Merge remote-tracking branch 'db/main' into iceberg/create-catalog

f5ff5b4

fix: sync with databend-meta

1e7c282

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ClSlaid added 3 commits February 20, 2023 13:01

ci: try fix sqllogictest error

3660cb7

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ci: try fix sqllogictest by using fs endpoint

da8368a

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ci: make sqllogictest tolerate insecure

a83da2b

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ClSlaid force-pushed the iceberg/create-catalog branch from 7ccb5e4 to a83da2b Compare February 20, 2023 08:17

ci: try fix compat test

8b42e9f

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

ClSlaid force-pushed the iceberg/create-catalog branch from 55e2fd2 to 8b42e9f Compare February 20, 2023 13:51

ci: remove create catalog test

9455172

This PR addes new syntax along with its implementations, resulting imcompatibility against old versions. The test should be resumed by following PRs under this topic Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Xuanwo requested review from BohuTANG, Xuanwo and sundy-li February 21, 2023 02:54

Xuanwo approved these changes Feb 21, 2023

View reviewed changes

Xuanwo marked this pull request as ready for review February 21, 2023 03:07

BohuTANG approved these changes Feb 21, 2023

View reviewed changes

mergify bot and others added 3 commits February 21, 2023 03:26

Merge branch 'main' into iceberg/create-catalog

6d6bb7d

Delete sqllogictest.log

f67fbff

Merge branch 'main' into iceberg/create-catalog

5b7d89e

vercel bot deployed to Preview February 21, 2023 03:39 View deployment

mergify bot merged commit 163b25d into databendlabs:main Feb 21, 2023

BohuTANG reviewed Feb 21, 2023

View reviewed changes

src/query/storages/iceberg/Cargo.toml Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Iceberg/create-catalog #9017

feat: Iceberg/create-catalog #9017

ClSlaid commented Nov 29, 2022 •

edited by BohuTANG

Loading

vercel bot commented Nov 29, 2022 •

edited

Loading

BohuTANG commented Dec 7, 2022

ClSlaid commented Dec 7, 2022

ClSlaid commented Feb 20, 2023

BohuTANG left a comment

feat: Iceberg/create-catalog #9017

feat: Iceberg/create-catalog #9017

Conversation

ClSlaid commented Nov 29, 2022 • edited by BohuTANG Loading

Summary

Example

Future Possibility

vercel bot commented Nov 29, 2022 • edited Loading

BohuTANG commented Dec 7, 2022

ClSlaid commented Dec 7, 2022

ClSlaid commented Feb 20, 2023

BohuTANG left a comment

Choose a reason for hiding this comment

ClSlaid commented Nov 29, 2022 •

edited by BohuTANG

Loading

vercel bot commented Nov 29, 2022 •

edited

Loading