Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException #3402

Closed
wants to merge 1 commit into from

Conversation

tsianglei
Copy link
Contributor

What is the purpose of the pull request

HUDI-2167
Fix org.apache.hudi.util.StreamerUtil#getHoodieClientConfig does not set HoodieCleaningPolicy but org.apache.hudi.config.HoodieCompactionConfig.Builder#build check the policy. Then will got NullPointerException.
Exception stack is:

Caused by: java.lang.NullPointerException: Name is null
	at java.lang.Enum.valueOf(Enum.java:236) ~[?:1.8.0_221]
	at org.apache.hudi.common.model.HoodieCleaningPolicy.valueOf(HoodieCleaningPolicy.java:24) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
	at org.apache.hudi.config.HoodieCompactionConfig$Builder.build(HoodieCompactionConfig.java:392) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
	at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:162) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
	at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:305) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:153) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:189) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.startAllOperatorCoordinators(SchedulerBase.java:1253) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:624) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
	at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:1032) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
	at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705) ~[?:1.8.0_221]

Brief change log

  • add HoodieCleaningPolicy

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot
Copy link

hudi-bot commented Aug 4, 2021

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run travis re-run the last Travis build
  • @hudi-bot run azure re-run the last Azure build

@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
.withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't HoodieCleaningPolicy.KEEP_LATEST_COMMITS the default value ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build() method get policy use compactionConfig.getString(CLEANER_POLICY_PROP),this method doesn't return default value, but null. if not set then will return null.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HoodieCompactionConfig would setDefaults first before build.

@vinothchandar vinothchandar added this to Ready for Review in PR Tracker Board Aug 10, 2021
@danny0405
Copy link
Contributor

Not a valid fix, close it.

@danny0405 danny0405 closed this Aug 11, 2021
PR Tracker Board automation moved this from Ready for Review to Done Aug 11, 2021
@danny0405 danny0405 reopened this Aug 18, 2021
PR Tracker Board automation moved this from Done to Under Discussion PRs Aug 18, 2021
@danny0405 danny0405 closed this in b7a0d76 Aug 18, 2021
PR Tracker Board automation moved this from Under Discussion PRs to Done Aug 18, 2021
a0x8o added a commit to a0x8o/hudi that referenced this pull request Aug 18, 2021
umehrot2 pushed a commit that referenced this pull request Aug 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

None yet

3 participants