Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-5977] Fix Date to String column schema evolution #8280

Merged
merged 6 commits into from
Mar 28, 2023

Conversation

voonhous
Copy link
Member

@voonhous voonhous commented Mar 24, 2023

Change Logs

Fix date to string schema evolution when vectorized readers are not used.

Issue and stacktrace can be found here:
https://issues.apache.org/jira/browse/HUDI-5977

Impact

Tables can be read after a date -> string schema evolution is performed and when vectorized readers are not used.

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

…to become unreadable when non-vectorized readers are used
@voonhous
Copy link
Member Author

@xiarixiaoyao Hello, can you please help to review the fixes here?

A test case is included to reproduce this issue + verify the fix.

Thank you.

@boundarymate
Copy link
Contributor

Hello,does this PR solve the problem mentioned in https://www2.jianshu.com/p/385483e3d58f?

@voonhous
Copy link
Member Author

Hello,does this PR solve the problem mentioned in https://www2.jianshu.com/p/385483e3d58f?

Erm, the issue that you've linked in the article seems to be an unrelated to the issue that is described in this PR.

@xiarixiaoyao xiarixiaoyao self-assigned this Mar 24, 2023
@codope codope changed the title [HUDI-5977] Fix Date to String column schema evolution causing table … [HUDI-5977] Fix Date to String column schema evolution Mar 24, 2023
@codope codope added schema-and-data-types priority:major degraded perf; unable to move forward; potential bugs spark-sql on-call-triaged labels Mar 24, 2023
@xiarixiaoyao
Copy link
Contributor

@voonhous
Thank you for your contribution, LGTM

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@xiarixiaoyao xiarixiaoyao merged commit 04ec593 into apache:master Mar 28, 2023
@voonhous voonhous deleted the HUDI-5977 branch March 28, 2023 16:53
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
stayrascal pushed a commit to stayrascal/hudi that referenced this pull request Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
on-call-triaged priority:major degraded perf; unable to move forward; potential bugs schema-and-data-types spark-sql
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

6 participants