Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-2014] Support flink hive sync in batch mode #3081

Merged
merged 1 commit into from Jun 16, 2021

Conversation

swuferhong
Copy link
Contributor

Tips

What is the purpose of the pull request

The item of this PR is to support flink hive sync in batch mode, while hive sync now just run in flink streaming mode. But, batch mode also need to do hive sync.

Brief change log

(for example:)

  • Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@codecov-commenter
Copy link

codecov-commenter commented Jun 15, 2021

Codecov Report

Merging #3081 (8432465) into master (769dd2d) will increase coverage by 2.59%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #3081      +/-   ##
============================================
+ Coverage     50.04%   52.63%   +2.59%     
+ Complexity     3685      407    -3278     
============================================
  Files           526       70     -456     
  Lines         25466     2880   -22586     
  Branches       2886      359    -2527     
============================================
- Hits          12744     1516   -11228     
+ Misses        11454     1220   -10234     
+ Partials       1268      144    -1124     
Flag Coverage Δ
hudicli ?
hudiclient ∅ <ø> (∅)
hudicommon ?
hudiflink ?
hudihadoopmr ?
hudisparkdatasource ?
hudisync 6.79% <ø> (-44.66%) ⬇️
huditimelineservice ?
hudiutilities 71.01% <ø> (+61.91%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../org/apache/hudi/hive/NonPartitionedExtractor.java 0.00% <0.00%> (-100.00%) ⬇️
...he/hudi/hive/HiveStylePartitionValueExtractor.java 0.00% <0.00%> (-100.00%) ⬇️
...main/java/org/apache/hudi/hive/HiveSyncConfig.java 0.00% <0.00%> (-97.83%) ⬇️
.../apache/hudi/hive/MultiPartKeysValueExtractor.java 0.00% <0.00%> (-90.91%) ⬇️
...in/java/org/apache/hudi/hive/SchemaDifference.java 0.00% <0.00%> (-84.85%) ⬇️
...in/java/org/apache/hudi/hive/util/ConfigUtils.java 0.00% <0.00%> (-73.92%) ⬇️
...in/java/org/apache/hudi/hive/HoodieHiveClient.java 0.00% <0.00%> (-71.61%) ⬇️
...va/org/apache/hudi/hive/util/ColumnNameXLator.java 0.00% <0.00%> (-70.00%) ⬇️
...java/org/apache/hudi/hive/util/HiveSchemaUtil.java 0.00% <0.00%> (-68.94%) ⬇️
...c/main/java/org/apache/hudi/hive/HiveSyncTool.java 0.00% <0.00%> (-67.55%) ⬇️
... and 501 more

@swuferhong swuferhong closed this Jun 15, 2021
@swuferhong swuferhong reopened this Jun 15, 2021
@yanghua yanghua self-assigned this Jun 15, 2021
@hudi-bot
Copy link

hudi-bot commented Jun 15, 2021

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run travis re-run the last Travis build
  • @hudi-bot run azure re-run the last Azure build

@vinothchandar vinothchandar added this to Ready for Review in PR Tracker Board Jun 16, 2021
PR Tracker Board automation moved this from Ready for Review to Nearing Landing Jun 16, 2021
Copy link
Contributor

@yanghua yanghua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yanghua yanghua merged commit 8b0a502 into apache:master Jun 16, 2021
PR Tracker Board automation moved this from Nearing Landing to Done Jun 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

None yet

4 participants