[HUDI-2277] HoodieDeltaStreamer reading ORC files directly using ORCDFSSource by zhangyue19921010 · Pull Request #3413 · apache/hudi

zhangyue19921010 · 2021-08-05T09:29:45Z

https://issues.apache.org/jira/projects/HUDI/issues/HUDI-2277

What is the purpose of the pull request

Develop a new Source named ORCDFSSource extended from RowSource

Now, HoodieDeltaStreamer can read orc files directly using ORCDFSSource.

Also add UTs which are necessary and tested on our local env.

Brief change log

(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hudi-bot · 2021-08-05T09:32:04Z

CI report:

a7a5955 UNKNOWN
47ca820 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

zhangyue19921010 · 2021-08-05T16:50:54Z

@hudi-bot run travis

vinothchandar

Just some small comments. We can land once they are addressed

vinothchandar · 2021-08-10T19:49:06Z

hudi-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java

    }
  }
+
+  public static void addAvroRecord(


can this sit somewhere in a test utils class? given its only used by a test?

Sure, Done.

vinothchandar · 2021-08-10T19:50:42Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

+  }
+
+  @Test
+  public void testORCDFSSourceWithSourceSchemaFileAndNoTransformer() throws Exception {


for sake of test runtime. could we test just 1-2 combos here. most of the testing done for row source (ParquetDFS) should cover already?

Okay, done. Thanks a lot for your review :)

zhangyue19921010 · 2021-09-06T02:49:39Z

Hi @vinothchandar sorry to bother you. Since this patch is passed all ut/it. So could you please take a look at your convince？ Thanks a lot!

vinothchandar · 2021-09-07T22:24:31Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

+            PROPS_FILENAME_TEST_ORC, ORC_SOURCE_ROOT, false);
+  }
+
+  private void prepareORCDFSSource(boolean useSchemaProvider, boolean hasTransformer, String sourceSchemaFile, String targetSchemaFile,


wonder if we can simplify this test?

Srue thing. done. thanks a lot for your review.

zhangyue19921010 · 2021-09-08T09:14:28Z

@hudi-bot run azure

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java

nsivabalan · 2021-09-25T15:28:17Z

@zhangyue19921010 : do ping me here once you have addressed all comments. I can take a look.

zhangyue19921010 · 2021-09-26T08:08:07Z

@zhangyue19921010 : do ping me here once you have addressed all comments. I can take a look.

Hi @nsivabalan Thanks a lot for your review. I finished all the changes. PTAL :)

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

zhangyue19921010 · 2021-09-28T06:04:32Z

Hi @nsivabalan @vinothchandar. Thanks a lot for your attention, review and approve! Could we land it or what else do I need to do? :)

xushiyan

LGTM. some minor code suggestions. Please go ahead and merge once fixed and Azure CI passed. Thanks.

...tilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamerBase.java

hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java

xushiyan

@zhangyue19921010 we're almost there :) just 1 more nitpick. thanks for making the feature.

hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java

zhangyue19921010 · 2021-09-29T09:07:53Z

Hi @xushiyan Thanks a lot for your attention and review. My bad for misunderstanding :) code changed and waiting for ci/cd green.

…FSSource (apache#3413) * add ORCDFSSource to support reading orc file into hudi format && add UTs * remove ununsed import * simplify tes * code review * code review * code review * code review * code review * code review Co-authored-by: yuezhang <yuezhang@freewheel.tv>

yuezhang added 2 commits August 4, 2021 14:39

add ORCDFSSource to support reading orc file into hudi format && add UTs

75f3cf2

Merge branch 'master' into OrcDFSSource

45fbd4f

zhangyue19921010 changed the title ~~[HUDI-2277] Let HoodieDeltaStreamer reading ORC files directly using ORCDFSSource~~ [HUDI-2277] HoodieDeltaStreamer reading ORC files directly using ORCDFSSource Aug 5, 2021

zhangyue19921010 closed this Aug 7, 2021

zhangyue19921010 reopened this Aug 7, 2021

vinothchandar approved these changes Aug 10, 2021

View reviewed changes

yuezhang added 4 commits August 11, 2021 16:21

code reviewed

f2c826d

remove ununsed import

cf91f8e

Merge branch 'master' into OrcDFSSource

83668ba

Merge branch 'master' into OrcDFSSource

2854654

vinothchandar assigned nsivabalan and xushiyan Sep 7, 2021

vinothchandar reviewed Sep 7, 2021

View reviewed changes

yuezhang added 2 commits September 8, 2021 15:12

simplify tes

d1cb1fb

Merge branch 'master' into OrcDFSSource

7a21d39

nsivabalan requested changes Sep 20, 2021

View reviewed changes

yuezhang added 4 commits September 22, 2021 10:14

code review

3c0d041

Merge branch 'master' into OrcDFSSource

3222314

code review

a7a5955

code review

99414ba

nsivabalan approved these changes Sep 28, 2021

View reviewed changes

hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java Show resolved Hide resolved

xushiyan approved these changes Sep 28, 2021

View reviewed changes

code review

48c1b57

xushiyan reviewed Sep 28, 2021

View reviewed changes

code review

9412978

xushiyan reviewed Sep 29, 2021

View reviewed changes

hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java Outdated Show resolved Hide resolved

hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java Outdated Show resolved Hide resolved

yuezhang added 2 commits September 29, 2021 17:02

code review

6471cf4

code review

47ca820

xushiyan merged commit dd1bd62 into apache:master Sep 29, 2021

zhangyue19921010 mentioned this pull request Oct 12, 2021

[Minor][asf-site] Modify hoodieDeltaStreamer user docs, adding more merged source-class #3785

Merged

5 tasks

Conversation

zhangyue19921010 commented Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

hudi-bot commented Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

zhangyue19921010 commented Aug 5, 2021

Uh oh!

vinothchandar left a comment

Choose a reason for hiding this comment

Uh oh!

vinothchandar Aug 10, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Aug 11, 2021

Choose a reason for hiding this comment

Uh oh!

vinothchandar Aug 10, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Aug 11, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Sep 6, 2021

Uh oh!

vinothchandar Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Sep 8, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nsivabalan commented Sep 25, 2021

Uh oh!

zhangyue19921010 commented Sep 26, 2021

Uh oh!

Uh oh!

zhangyue19921010 commented Sep 28, 2021

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangyue19921010 commented Sep 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhangyue19921010 commented Aug 5, 2021 •

edited

Loading

hudi-bot commented Aug 5, 2021 •

edited

Loading