Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EPIC] Redesign Graviton's Schema and Connector System #43

Closed
2 tasks done
jerryshao opened this issue Jun 6, 2023 · 0 comments
Closed
2 tasks done

[EPIC] Redesign Graviton's Schema and Connector System #43

jerryshao opened this issue Jun 6, 2023 · 0 comments
Assignees
@jerryshao
Labels
epic Key feature
Milestone
Graviton v0.1.0: ...

Comments

@jerryshao
Copy link
Collaborator

jerryshao commented Jun 6, 2023 •
edited

Describe the proposal

Current Graviton's Schema system follows "lakehouse/zone/table" logical structure, while the mapping physical table has different physical structure "metasouce/db/table", it's quite confuse for user to understand these two metadata structures, also it's hard to manage the metadata. So we decide to redesign the schema system to make it simpler and easy to understand.

The details of thinking can be found here (https://docs.google.com/document/d/1Wrd9HHJF2wLDhsvKix1VVKWLi_RdDhw-X3rqg-4lwtw/edit#heading=h.4adypg9kzjky).

Task list
@jerryshao jerryshao self-assigned this Jun 6, 2023
@jerryshao jerryshao added the epic Key feature label Jun 6, 2023
This was referenced Jun 6, 2023
Closed
Merged
Closed
jerryshao added a commit that referenced this issue Jun 14, 2023
@jerryshao
[#44] refactor: refactor the schema system and add catalog interfaces (…
49d4add 
…#45)

### What changes were proposed in this pull request?

This PR changes include:

1. refactors the whole metadata system, which simplifies the current
tenant/lakehouse/zone/table schema structure to
lakehouse/catalog/<meta-structure> (refer to what metacat did).
2. Add catalog interfaces and define a series of catalog behaviors. the
specific catalog implementation could inherit these interfaces to
achieve its own one (refer to what Spark connector/catalog did).
3. rearchitect the code structure.

### Why are the changes needed?

As #43 described, the current schema system is a bit difficult to manage
(to pursue virtual semantics), we needs to simplify the current design.

Fix: #44

### Does this PR introduce _any_ user-facing change?

We're still in the early stage of the project, the change is unavoidable
and acceptable.

### How was this patch tested?

UTs to cover the codes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jerryshao jerryshao

Labels
epic Key feature
Projects
None yet
Milestone
Graviton v0.1.0: Implement a single-s...
Development

No branches or pull requests
1 participant
@jerryshao