Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARCTIC-1288][ams] limit the total files size bytes for Optimizing #1294

Merged
merged 11 commits into from
Apr 11, 2023

Conversation

wangtaohz
Copy link
Contributor

Why are the changes needed?

fix #1288

Brief change log

  • add table property self-optimizing.max-file-size-bytes with default value 8 GB
  • optimizing plan with max file size bytes limit, and skipping some partitions
  • Mixed format table's Minor-Optimizing only calculates the file size of the Change Store
  • Iceberg format table's Minor-Optimizing only calculates the file size of small files/pos-delete files/equ-delete files

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:ams-server Ams server module module:core Core module module:ams-dashboard Ams dashboard module type:docs Improvements or additions to documentation labels Mar 28, 2023
@codecov
Copy link

codecov bot commented Mar 28, 2023

Codecov Report

Patch coverage: 87.50% and project coverage change: -0.01 ⚠️

Comparison is base (9dc17c5) 29.51% compared to head (4cfede9) 29.51%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1294      +/-   ##
============================================
- Coverage     29.51%   29.51%   -0.01%     
- Complexity     5410     5414       +4     
============================================
  Files           695      695              
  Lines         71169    71178       +9     
  Branches       8185     8185              
============================================
+ Hits          21008    21010       +2     
- Misses        48147    48154       +7     
  Partials       2014     2014              
Flag Coverage Δ
core 28.23% <87.50%> (-0.01%) ⬇️
trino 52.79% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...java/com/netease/arctic/table/TableProperties.java 0.00% <ø> (ø)
...ctic/ams/server/optimize/AbstractOptimizePlan.java 86.31% <85.71%> (-1.19%) ⬇️
.../ams/server/optimize/IcebergMinorOptimizePlan.java 91.89% <100.00%> (+0.07%) ⬆️
.../arctic/ams/server/optimize/MinorOptimizePlan.java 85.24% <100.00%> (+0.24%) ⬆️

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@wangtaohz wangtaohz merged commit 100b25e into apache:master Apr 11, 2023
@wangtaohz wangtaohz deleted the fix-1288 branch April 11, 2023 01:46
wangtaohz added a commit that referenced this pull request Apr 13, 2023
…1294)

* add self-optimizing.max-file-size-bytes

* add test case

* modify doc

* refactor method name: reachOptimizeLimit

---------

Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
(cherry picked from commit 100b25e)
Signed-off-by: wangtao <wangtao3@corp.netease.com>
@wangtaohz
Copy link
Contributor Author

cherry-picked to 0.4.x

zhoujinsong added a commit that referenced this pull request May 31, 2023
…1294)

* add self-optimizing.max-file-size-bytes

* add test case

* modify doc

* refactor method name: reachOptimizeLimit

---------

Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…pache#1294)

* add self-optimizing.max-file-size-bytes

* add test case

* modify doc

* refactor method name: reachOptimizeLimit

---------

Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:ams-dashboard Ams dashboard module module:ams-server Ams server module module:core Core module type:docs Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement]: Full-Optimizing occupies all optimizer resources making other tables' Self-Optimizing hang up
2 participants