Skip to content

[HUDI-7201] Schema Evolution: use target schema if source is empty#10288

Merged
nsivabalan merged 2 commits intoapache:masterfrom
jonvex:fix_empty_commit_schema_evolution
Dec 10, 2023
Merged

[HUDI-7201] Schema Evolution: use target schema if source is empty#10288
nsivabalan merged 2 commits intoapache:masterfrom
jonvex:fix_empty_commit_schema_evolution

Conversation

@jonvex
Copy link
Contributor

@jonvex jonvex commented Dec 8, 2023

Change Logs

If the incoming df is empty, the commit schema will be empty. Now, it will be the current table schema.

Impact

Currently if a user does an empty commit, we won't be able to detect invalid schema evolution in the next commit

Risk level (write none, low medium or high below)

low

Documentation Update

N/A

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@jonvex jonvex marked this pull request as ready for review December 8, 2023 18:20
@jonvex jonvex changed the title [HUDI-7201] use target schema if source is empty [HUDI-7201] Schema Evolution: use target schema if source is empty Dec 8, 2023
@hudi-bot
Copy link
Collaborator

hudi-bot commented Dec 8, 2023

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@nsivabalan nsivabalan added priority:critical Production degraded; pipelines stalled release-0.14.1 labels Dec 10, 2023
Copy link
Contributor

@bvaradar bvaradar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense

@nsivabalan
Copy link
Contributor

image

@nsivabalan nsivabalan merged commit 5fb00ef into apache:master Dec 10, 2023
@zyclove
Copy link

zyclove commented Dec 11, 2023

@nsivabalan @danny0405
Seeing that there are still 34 issues to be fixed in release 0.14.1, we are very looking forward to it.
We hope it can be speeded up. We also hope to switch the prod business to version 0.14.1 as soon as possible.
It is best to be done before the year end. Version 0.14.1 should be the best and most complete version ever.
I am also looking forward to the official version 1.0. thanks everyone.

nsivabalan pushed a commit to nsivabalan/hudi that referenced this pull request Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:critical Production degraded; pipelines stalled release-0.14.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants