[HUDI-5309] Support for Spark to automatically enable schema evolution when reading the Hoodie table#7301
Conversation
…ng the Hoodie table
fuyun2024
changed the title
fuyun2024
changed the title
|
@alexeykudinkin Do you have time to help review this PR? |
| None | ||
| } | ||
| val internalSchemaOpt = Try { | ||
| specifiedQueryTimestamp.map(schemaResolver.getTableInternalSchemaFromCommitMetadata) |
There was a problem hiding this comment.
@fuyun2024 Can you please create a JIRA for this change? This does not look like a minor change.
Here, commit metadata will always be scanned. Previously, it would be scanned only when schema evolution was enabled.
There was a problem hiding this comment.
@fuyun2024 unfortunately we can't do that -- @nsivabalan is actually currently working on the opposite change that will make sure that Schema Evolution is guarded behind the flag such that we can force it disabled if needs to be (even though it might not be safe for some use-cases, there are cases when this still is necessary)
codope
added
area:schema
fuyun2024
changed the title
|
@YannByron @xushiyan : is this safe to flip? can either of you review this please. |
I agree that enable schema evolution without any config. Looks good. |
alexeykudinkin
left a comment
alexeykudinkin
left a comment
There was a problem hiding this comment.
@nsivabalan to chime in, since you have a PR doing the opposite
| None | ||
| } | ||
| val internalSchemaOpt = Try { | ||
| specifiedQueryTimestamp.map(schemaResolver.getTableInternalSchemaFromCommitMetadata) |
There was a problem hiding this comment.
@fuyun2024 unfortunately we can't do that -- @nsivabalan is actually currently working on the opposite change that will make sure that Schema Evolution is guarded behind the flag such that we can force it disabled if needs to be (even though it might not be safe for some use-cases, there are cases when this still is necessary)
|
Thanks for your time, I will close this PR. |
6 participants
Support for Spark to automatically enable schema evolution when reading the Hoodie table.
Change Logs
Check if table has internal schema to enable schema evolution.
Impact
none
Risk level (write none, low medium or high below)
none
Documentation Update
No doc or configuration changed.
Contributor's checklist