Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support automatically generating and managing Tags and Branchs for Table #1354

Open
1 of 2 tasks
Tracked by #2176 ...
zhoujinsong opened this issue Apr 17, 2023 · 0 comments
Open
1 of 2 tasks
Tracked by #2176 ...
Labels
type:feature Feature Requests

Comments

@zhoujinsong
Copy link
Contributor

zhoujinsong commented Apr 17, 2023

Description

Tags and branches are common concepts for managing multiple versions of data. Apache Iceberg has supported tags and branches since version 0.14, and the functionality has been improved in version 1.2.0. Using tags and branches can achieve more sophisticated management of snapshot lifecycles, addressing the following scenarios:

  • Long-term Time Travel: fulfilling the audit needs of long-term historical data
  • WAP (Write Audit Publish) writing: supporting audit the written data before formal publishing.

Use case/motivation

The Tag/Branch management functionality of Arctic is expected to provide the following capabilities on top of Iceberg:

  • More user-friendly: including management of tags/branches on Web UI, automatic creation, and lifecycle management
  • More accurate: correcting data on the branch, generating more precise tags (Iceberg can currently only generate imprecise tags/branches on discrete snapshots, and the accuracy depends on the granularity of the snapshot commits)
  • More universal: supporting tags/branches on multiple table formats under Arctic.

Describe the solution

Refer to the documentation for detailed design.
https://docs.google.com/document/d/1_56YHZO7XSkkZV7bTFQ7yWNVfiV0TknwOIMt0j8Me6Y/edit?usp=sharing

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@zhoujinsong zhoujinsong added the type:feature Feature Requests label Apr 17, 2023
@zhoujinsong zhoujinsong mentioned this issue Apr 17, 2023
4 tasks
@wangtaohz wangtaohz changed the title [Feature]: Support generating a new partition containing full data every day for Hive users [Feature]: Support automatically generating and managing tags for Table Jul 24, 2023
@wangtaohz wangtaohz changed the title [Feature]: Support automatically generating and managing tags for Table [Feature]: Support automatically generating and managing Tags and Branchs for Table Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:feature Feature Requests
Projects
None yet
Development

No branches or pull requests

1 participant