Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ARCTIC-1863]Support optimizing for partition evolution iceberg table #2303

Merged
merged 19 commits into from
Nov 28, 2023
Merged

[ARCTIC-1863]Support optimizing for partition evolution iceberg table #2303

merged 19 commits into from
Nov 28, 2023

Conversation

shidayang
Copy link
Contributor

@shidayang shidayang commented Nov 14, 2023 •
edited
Loading

Why are the changes needed?

Close #1863 .

Brief change log

  • Modify com.netease.arctic.server.dashboard.MixedAndIcebergTableDescriptor#collectFileInfo to fix dashboard display error
  • Fix OptimizingPlaner error when planning partition evolution iceberg table
  • Modify IcebergRewriteExecutor to write insert file in new spec and write position delete in old spec
  • Modify IcebergRewriteExecutor to replace FileWriter by TaskWriter for write multiple partition files

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:core Core module module:mixed-hive Hive moduel for Mixed Format module:ams-dashboard Ams dashboard module labels Nov 14, 2023
@shidayang
Merge remote-tracking branch 'origin/master' into partition-evalution
0e804d9 
# Conflicts:
#	ams/server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java
@shidayang
merge master
72052bb
@codecov Codecov
Copy link

codecov bot commented Nov 14, 2023 •
edited
Loading

Codecov Report

Attention: 134 lines in your changes are missing coverage. Please review.

Comparison is base (f11336f) 52.82% compared to head (27bf85c) 52.93%.
Report is 1 commits behind head on master.

Files Patch % Lines
...rver/dashboard/MixedAndIcebergTableDescriptor.java 0.00% 60 Missing ⚠️
...main/java/org/apache/iceberg/IcebergFindFiles.java 71.50% 34 Missing and 17 partials ⚠️
...ctic/hive/optimizing/MixFormatRewriteExecutor.java 0.00% 9 Missing ⚠️
...ctic/server/dashboard/model/PartitionBaseInfo.java 0.00% 5 Missing ⚠️
...ease/arctic/optimizing/IcebergRewriteExecutor.java 70.00% 2 Missing and 1 partial ⚠️
...c/server/dashboard/controller/TableController.java 0.00% 2 Missing ⚠️
.../server/dashboard/model/PartitionFileBaseInfo.java 0.00% 2 Missing ⚠️
...arctic/server/dashboard/ServerTableDescriptor.java 0.00% 1 Missing ⚠️
...ic/server/optimizing/plan/OptimizingEvaluator.java 80.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##             master    #2303      +/-   ##
============================================
+ Coverage     52.82%   52.93%   +0.11%     
- Complexity     4262     4296      +34     
============================================
  Files           514      516       +2     
  Lines         29521    29698     +177     
  Branches       2867     2894      +27     
============================================
+ Hits          15594    15722     +128     
- Misses        12665    12698      +33     
- Partials       1262     1278      +16
Flag Coverage Δ
core 53.31% <53.79%> (+0.17%) ⬆️
trino 50.93% <ø> (-0.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@shidayang
Merge remote-tracking branch 'origin/master' into partition-evalution
e7eaf47 
# Conflicts:
#	ams/dashboard/src/main/resources/static/css/chunk-9fc4147c.d250cc66.css
#	ams/dashboard/src/main/resources/static/css/chunk-cc7f866c.3e429678.css
#	ams/dashboard/src/main/resources/static/css/chunk-f8e3eb7e.30a759b2.css
#	ams/dashboard/src/main/resources/static/index.html
#	ams/dashboard/src/main/resources/static/js/chunk-28b9a9cc.8ad4d357.js
#	ams/dashboard/src/main/resources/static/js/chunk-3900a95a.39f8ad9f.js
#	ams/dashboard/src/main/resources/static/js/chunk-vendors.6eab6434.js
#	ams/server/src/main/java/com/netease/arctic/server/utils/IcebergTableUtil.java
@shidayang
Fix conflict
ed395ee
wangtaohz
wangtaohz reviewed Nov 22, 2023
View reviewed changes
ams/server/pom.xml Outdated Show resolved Hide resolved
wangtaohz
wangtaohz reviewed Nov 22, 2023
View reviewed changes
Copy link
Contributor

@wangtaohz wangtaohz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left some comments, PTAL. @shidayang

ams/server/src/main/java/com/netease/arctic/server/dashboard/PaimonTableDescriptor.java Show resolved Hide resolved
ams/server/src/main/java/com/netease/arctic/server/dashboard/model/PartitionBaseInfo.java Outdated Show resolved Hide resolved
ams/server/src/main/java/com/netease/arctic/server/dashboard/model/PartitionFileBaseInfo.java Show resolved Hide resolved
core/src/test/java/com/netease/arctic/optimizing/IcebergRewriteExecutorTest.java Outdated Show resolved Hide resolved
core/src/test/java/com/netease/arctic/optimizing/IcebergRewriteExecutorTest.java Show resolved Hide resolved
core/src/main/java/org/apache/iceberg/IcebergFindFiles.java Show resolved Hide resolved
...server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java Outdated Show resolved Hide resolved
@CLAassistant
Copy link

CLAassistant commented Nov 22, 2023 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

wangtaohz
wangtaohz approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@wangtaohz wangtaohz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

There are some conflicts in the front-end static code that need to be resolved.

zhoujinsong
zhoujinsong reviewed Nov 27, 2023
View reviewed changes
Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shidayang I left some small suggestion.
Please task another look.

core/src/main/java/org/apache/iceberg/IcebergFindFiles.java Outdated Show resolved Hide resolved
...server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java Outdated Show resolved Hide resolved
...server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java Outdated Show resolved Hide resolved
zhoujinsong
zhoujinsong approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@shidayang shidayang merged commit 15fff6f into apache:master Nov 28, 2023
7 checks passed
@shidayang shidayang deleted the partition-evalution branch November 28, 2023 11:32
hameizi pushed a commit to hameizi/arctic that referenced this pull request Nov 30, 2023
@shidayang @hameizi
[ARCTIC-1863]Support optimizing for partition evolution iceberg table (…
0829bb4 
…apache#2303)

(cherry picked from commit 15fff6f)
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
@shidayang @ShawHee
[ARCTIC-1863]Support optimizing for partition evolution iceberg table (…
9ce17d7 
…apache#2303)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangtaohz wangtaohz wangtaohz approved these changes

@zhoujinsong zhoujinsong zhoujinsong approved these changes

Assignees
No one assigned
Labels
module:ams-dashboard Ams dashboard module module:core Core module module:mixed-hive Hive moduel for Mixed Format
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Iceberg mixed partition table, amoro UI display and merged write directory error
4 participants
@shidayang @CLAassistant @zhoujinsong @wangtaohz