Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]Feature optimize support hive #245

Merged
merged 68 commits into from
Sep 1, 2022

Conversation

hzluting
Copy link
Contributor

Why are the changes needed?

optimize support hive #38

Brief change log

  • Add the 'FullMajor' optimize type to optimize whole arctic base store
  • Add the 'optimize.full.trigger.max-interval' configuration to trigger FullMajor
  • For KeyedTable FullMajor trigger by small file size and trigger interval, the UnKeyedTable only trigger by trigger interval
  • Now, Major optimize commit process is overwrite data files first, then try to delete delete files
  • Now, Minor optimize isn't support hive. For The KeyedTable FullMajor and when Major with no position-delete will support hive. For the UnKeyedTable FullMajor and Major all support hive
  • Add a thread pool to sync iceberg metadata to hive metadata

How was this patch tested?

  • use support hive catalog

luting and others added 30 commits July 27, 2022 11:35
…apt-hive

# Conflicts:
#	core/src/main/java/com/netease/arctic/io/writer/GenericTaskWriters.java
luting added 3 commits August 30, 2022 17:02
…apt-hive

# Conflicts:
#	ams/ams-server/src/test/java/com/netease/arctic/ams/server/AmsTestBase.java
#	ams/ams-server/src/test/java/com/netease/arctic/ams/server/OutAmsTestBase.java
hzluting and others added 21 commits August 31, 2022 10:24
…apt-hive

# Conflicts:
#	ams/ams-server/src/main/resources/sql/derby/ams-init.sql
#	ams/ams-server/src/main/resources/sql/mysql/upgrade-0.3.0-to-0.3.1.sql
…apt-hive

# Conflicts:
#	ams/ams-server/src/main/java/com/netease/arctic/ams/server/utils/ThreadPool.java
#	ams/ams-server/src/test/java/com/netease/arctic/ams/server/AmsTestBase.java
…apt-hive

# Conflicts:
#	ams/ams-server/pom.xml
#	ams/ams-server/src/main/java/com/netease/arctic/ams/server/service/ServiceContainer.java
#	ams/ams-server/src/test/java/com/netease/arctic/ams/server/AmsTestBase.java
#	hive/src/main/java/com/netease/arctic/hive/utils/HivePartitionUtil.java
@wangtaohz wangtaohz merged commit f0e6611 into apache:master Sep 1, 2022
@hzluting hzluting deleted the feature-optimize-adapt-hive branch September 1, 2022 06:01
@zhoujinsong zhoujinsong linked an issue Sep 6, 2022 that may be closed by this pull request
zhoujinsong pushed a commit that referenced this pull request May 31, 2023
* add rewrite to delete old position-delete

* remove pos-delete record count limit

* optimize support adapt hive

* optimize support hive and add full major optimize

* sync iceberg partition location to hive partition location

* adapt unit test to new struct

* add keyed unPartition table unit test

* add keyed unPartition table unit test

* merge hive sync arctic

* fix code review

* fix check style

* optimize unit test

* optimize code

* add latest full optimize time

* add update sql

* fix check style

* add file clean support hive

* fix unKeyed Full Major

* fix add twice

* fix check style

Co-authored-by: luting <ting.lt@dtwave-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Tracer] Empty Snapshot is lost
3 participants