Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-2590] Adding tests to validate different key generators #4473

Merged
merged 1 commit into from Jan 4, 2022

Conversation

nsivabalan
Copy link
Contributor

@nsivabalan nsivabalan commented Dec 30, 2021

What is the purpose of the pull request

Added tests to validate COW table for different queries for different key generators.
this is a redo of #3877

Brief change log

(for example:)

  • Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

Copy link
Member

@codope codope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Let me know what you think of the comment below.

var inputDF2 = spark.read.json(spark.sparkContext.parallelize(records2, 2))

if (classOf[TimestampBasedKeyGenerator].getName.equals(keyGenClass)) {
// incase of Timestamp based key gen, current_ts should not be updated. but dataGen.generateUpdates() would have updated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this was the issue with the test datagen. Would special handling of timestamp keygen in the datagen itself be better than doing it here in a test case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did take a stab, looks like we have to touch lot of methods in HoodieTestDataGenerator. Will take it as a follow up https://issues.apache.org/jira/browse/HUDI-3152

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Will land this.

@vinothchandar vinothchandar added this to Ready for Review in PR Tracker Board Jan 4, 2022
PR Tracker Board automation moved this from Ready for Review to Nearing Landing Jan 4, 2022
@codope codope merged commit 7329d22 into apache:master Jan 4, 2022
PR Tracker Board automation moved this from Nearing Landing to Done Jan 4, 2022
@vinishjail97 vinishjail97 mentioned this pull request Jan 5, 2022
5 tasks
@vinishjail97 vinishjail97 mentioned this pull request Jan 24, 2022
5 tasks
liusenhua pushed a commit to liusenhua/hudi that referenced this pull request Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

None yet

3 participants