[HUDI-1381] Schedule compaction based on time elapsed #2260

Karl-WangSK · 2020-11-18T08:05:50Z

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

Schedule compaction based on time elapsed

Brief change log

GH : #2229
It would be helpful to introduce configuration to schedule compaction based on time elapsed since last scheduled compaction.

Verify this pull request

This pull request is already covered by existing tests, such as (please describe tests).

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

codecov-io · 2020-11-18T08:49:12Z

Codecov Report

Merging #2260 (b34c416) into master (b593f10) will increase coverage by 19.24%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #2260       +/-   ##
=============================================
+ Coverage     50.27%   69.51%   +19.24%     
+ Complexity     2990      357     -2633     
=============================================
  Files           410       53      -357     
  Lines         18406     1929    -16477     
  Branches       1885      229     -1656     
=============================================
- Hits           9253     1341     -7912     
+ Misses         8395      456     -7939     
+ Partials        758      132      -626

Flag	Coverage Δ	Complexity Δ
hudicli	`?`	`?`
hudiclient	`?`	`?`
hudicommon	`?`	`?`
hudiflink	`?`	`?`
hudihadoopmr	`?`	`?`
hudisparkdatasource	`?`	`?`
hudisync	`?`	`?`
huditimelineservice	`?`	`?`
hudiutilities	`69.51% <ø> (-0.14%)`	`0.00 <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ	Complexity Δ
...common/table/view/FileSystemViewStorageConfig.java
...di-cli/src/main/java/org/apache/hudi/cli/Main.java
...java/org/apache/hudi/common/util/ParquetUtils.java
...rg/apache/hudi/common/fs/NoOpConsistencyGuard.java
...e/hudi/common/table/log/HoodieLogFormatWriter.java
...spark3/internal/HoodieDataSourceInternalTable.java
...sioning/clean/CleanMetadataV1MigrationHandler.java
.../org/apache/hudi/hive/NonPartitionedExtractor.java
.../main/scala/org/apache/hudi/HoodieSparkUtils.scala
...g/apache/hudi/hadoop/HoodieParquetInputFormat.java
... and 351 more

Karl-WangSK · 2020-11-28T12:55:05Z

@n3nash @vinothchandar hi ,can you pls take a look at this pr?

yanghua · 2020-11-28T13:45:41Z

@wangxianghu Can you review this PR?

yanghua

Have a quick look, found two minor issues.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java

...di-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestInlineCompaction.java

wangxianghu

Hi @Karl-WangSK Thanks for your contribution, I left some comments you can consider

...lient/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

wangxianghu

@Karl-WangSK Thanks for addressing my concern, LGTM now!
cc @yanghua

yanghua

@Karl-WangSK Left some comments.

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

...di-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestInlineCompaction.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

Karl-WangSK · 2021-01-18T06:34:32Z

@yanghua ready to merge

yanghua · 2021-01-18T06:36:25Z

@yanghua ready to merge

thanks for your reminder, will review it again~

yanghua

@Karl-WangSK Left some comments.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

yanghua

@Karl-WangSK Left some comments.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java

...lient/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactType.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java

yanghua

@Karl-WangSK Left some comments.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

...di-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestInlineCompaction.java

hudi-common/src/main/java/org/apache/hudi/exception/HoodieCompactException.java

Karl-WangSK · 2021-01-26T03:17:34Z

cc @yanghua

vinothchandar

LGTM overall. Minor comments.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java

...ent-common/src/main/java/org/apache/hudi/table/action/compact/CompactionTriggerStrategy.java

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java

vinothchandar

LGTM

Karl-WangSK force-pushed the HUDI-1381 branch from 4e81a88 to 679310a Compare November 18, 2020 15:18

n3nash self-assigned this Nov 20, 2020

vinothchandar assigned bvaradar Nov 20, 2020

yanghua requested changes Nov 28, 2020

View reviewed changes

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java Outdated Show resolved Hide resolved

...di-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestInlineCompaction.java Outdated Show resolved Hide resolved

Karl-WangSK requested a review from yanghua November 28, 2020 14:40

Karl-WangSK force-pushed the HUDI-1381 branch from 0750f24 to 6dac804 Compare December 17, 2020 10:12

wangxianghu requested changes Dec 25, 2020

View reviewed changes

Karl-WangSK force-pushed the HUDI-1381 branch 2 times, most recently from 5fcea7d to 87a7489 Compare December 26, 2020 12:05

wangxianghu reviewed Dec 29, 2020

View reviewed changes

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java Outdated Show resolved Hide resolved

wangxianghu approved these changes Dec 29, 2020

View reviewed changes

yanghua reviewed Jan 6, 2021

View reviewed changes

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java Outdated Show resolved Hide resolved

Karl-WangSK force-pushed the HUDI-1381 branch from 04a88c1 to ee958de Compare January 7, 2021 02:09

Karl-WangSK closed this Jan 7, 2021

Karl-WangSK force-pushed the HUDI-1381 branch from ee958de to b593f10 Compare January 7, 2021 02:33

update

ddf8fca

Karl-WangSK reopened this Jan 7, 2021

wangxianghu reviewed Jan 8, 2021

View reviewed changes

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java Outdated Show resolved Hide resolved

update

d087f09

wangxianghu reviewed Jan 13, 2021

View reviewed changes

...rc/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java Outdated Show resolved Hide resolved

update

7d0453e

yanghua reviewed Jan 18, 2021

View reviewed changes

Karl-WangSK added 2 commits January 18, 2021 15:34

update

fbd87b4

add log in each case

96e596a

yanghua reviewed Jan 18, 2021

View reviewed changes

Karl-WangSK added 6 commits January 19, 2021 00:32

update

a74ea81

Update HoodieWriteConfig.java

4259478

update

3c3e12e

Merge branch 'HUDI-1381' of github.com:Karl-WangSK/hudi into HUDI-1381

a318c91

update

b9a1a63

update

d720b4e

wangxianghu reviewed Jan 19, 2021

View reviewed changes

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java Outdated Show resolved Hide resolved

Update HoodieCompactionConfig.java

48f7392

yanghua requested changes Jan 20, 2021

View reviewed changes

update

c2a695a

Karl-WangSK requested a review from yanghua January 23, 2021 07:35

vinothchandar assigned vinothchandar and unassigned bvaradar Jan 29, 2021

vinothchandar reviewed Feb 3, 2021

View reviewed changes

Karl-WangSK added 2 commits February 5, 2021 15:40

update

1ffe0f6

update

b34c416

vinothchandar approved these changes Feb 17, 2021

View reviewed changes

vinothchandar merged commit 9431aab into apache:master Feb 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-1381] Schedule compaction based on time elapsed #2260

[HUDI-1381] Schedule compaction based on time elapsed #2260

Karl-WangSK commented Nov 18, 2020

codecov-io commented Nov 18, 2020 •

edited

Loading

Karl-WangSK commented Nov 28, 2020

yanghua commented Nov 28, 2020

yanghua left a comment

wangxianghu left a comment

wangxianghu left a comment

yanghua left a comment

Karl-WangSK commented Jan 18, 2021

yanghua commented Jan 18, 2021

yanghua left a comment

yanghua left a comment

yanghua left a comment

Karl-WangSK commented Jan 26, 2021

vinothchandar left a comment

vinothchandar left a comment

[HUDI-1381] Schedule compaction based on time elapsed #2260

[HUDI-1381] Schedule compaction based on time elapsed #2260

Conversation

Karl-WangSK commented Nov 18, 2020

Tips

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

codecov-io commented Nov 18, 2020 • edited Loading

Codecov Report

Karl-WangSK commented Nov 28, 2020

yanghua commented Nov 28, 2020

yanghua left a comment

Choose a reason for hiding this comment

wangxianghu left a comment

Choose a reason for hiding this comment

wangxianghu left a comment

Choose a reason for hiding this comment

yanghua left a comment

Choose a reason for hiding this comment

Karl-WangSK commented Jan 18, 2021

yanghua commented Jan 18, 2021

yanghua left a comment

Choose a reason for hiding this comment

yanghua left a comment

Choose a reason for hiding this comment

yanghua left a comment

Choose a reason for hiding this comment

Karl-WangSK commented Jan 26, 2021

vinothchandar left a comment

Choose a reason for hiding this comment

vinothchandar left a comment

Choose a reason for hiding this comment

codecov-io commented Nov 18, 2020 •

edited

Loading