feat(schema): Migrate SchemaProviders in Hudi-Utilities to use HoodieSchema by the-other-tim-brown · Pull Request #14364 · apache/hudi

the-other-tim-brown · 2025-11-26T15:43:07Z

Describe the issue this Pull Request addresses

Migrates the SchemaProviders to provide the source and target schemas as HoodieSchema instead of Avro.
Addresses: #14276

Summary and Changelog

Update the method signatures for SchemaProvider to output HoodieSchema instead of the Avro schema
Update the SchemaPostProcessor to operate on HoodieSchema instead of the Avro schema
Added utilities to aide in testing and updated relevant test cases

Impact

Moves the repo towards operating entirely on HoodieSchema which will allow us to expand the types that Hudi supports and other schema related features

Risk Level

Low

Documentation Update

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaProvider.java

voonhous · 2025-12-03T07:49:08Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java

    this.formatAdapter = new SourceFormatAdapter(source, this.errorTableWriter, Option.of(props));

-    Supplier<Option<Schema>> schemaSupplier = schemaProvider == null ? Option::empty : () -> Option.ofNullable(schemaProvider.getSourceSchema());
+    Supplier<Option<Schema>> schemaSupplier = schemaProvider == null ? Option::empty : () -> Option.ofNullable(schemaProvider.getSourceHoodieSchema().toAvroSchema());


IIUC, schemaProvider#getSourceHoodieSchema() might return null. potential NPE (base class implementation) here?

Suggested change:

() -> Option.ofNullable(schemaProvider.getSourceHoodieSchema()).map(HoodieSchema::toAvroSchema);

Updating this

voonhous · 2025-12-03T07:51:12Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceFormatAdapter.java

-    MercifulJsonConverter.clearCache(inputBatch.getSchemaProvider().getSourceSchema().getFullName());
-    AvroConvertor convertor = new AvroConvertor(inputBatch.getSchemaProvider().getSourceSchema(), isFieldNameSanitizingEnabled(), getInvalidCharMask());
+    MercifulJsonConverter.clearCache(inputBatch.getSchemaProvider().getSourceHoodieSchema().getFullName());
+    AvroConvertor convertor = new AvroConvertor(inputBatch.getSchemaProvider().getSourceHoodieSchema().toAvroSchema(), isFieldNameSanitizingEnabled(), getInvalidCharMask());


Potential NPE here too.

The AvroConverter assumes that the schema is non-null today

bvaradar · 2025-12-03T22:36:47Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java

    this.formatAdapter = new SourceFormatAdapter(source, this.errorTableWriter, Option.of(props));

-    Supplier<Option<Schema>> schemaSupplier = schemaProvider == null ? Option::empty : () -> Option.ofNullable(schemaProvider.getSourceSchema());
+    Supplier<Option<Schema>> schemaSupplier = schemaProvider == null ? Option::empty : () -> Option.ofNullable(schemaProvider.getSourceHoodieSchema()).map(HoodieSchema::toAvroSchema);


I guess changing schemaSupplier to Supplier<Option> is in subsequent PR.

Yes, do we have a ticket for the StreamSync code or should I make one?

I don't think I added it. Can you kindly add it. thanks

#17480 -added this

hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaPostProcessor.java

bvaradar · 2025-12-03T23:11:55Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/FilebasedSchemaProvider.java


  @Override
-  public Schema getSourceSchema() {
+  public HoodieSchema getSourceHoodieSchema() {


While, we are the handling external implementation of SchemaProvider correctly, we are not preventing breakages when users extend from these concrete implementations which unfortunately is present. If users extend FilebasedSchemaProvider and override "public Schema getTargetSchema", then their implementation will silently not be used.

I guess it is safe to retain the existing implementation of getSourceSchema and getTargetSchema in each implementation of SchemaProvider with deprecated flag and remove in subsequent release.

That's a good point. Should we just delegate for now on the actual implementations? We can have the integration point with the rest of the code be through HoodieSchema while keeping these calls delegate to the older methods.

Yes, that makes sense 👍

… don't have failures on upgrade to 1.2.o

bvaradar

LGTM

the-other-tim-brown · 2025-12-04T18:15:32Z

Azure CI is passing but status is not updated

hudi-bot · 2025-12-04T18:15:42Z

CI report:

6cf6477 UNKNOWN
014de83 UNKNOWN
0ed42d1 UNKNOWN
90d5891 UNKNOWN
7d81d49 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

…Schema (apache#14364) * start migrating schema provider classes * make code compile * fix test setup * fix ExampleDataSchemaProvider * fix integ-test module * handle null schema in deduce schema step * fix getField and default value handling, fix test setups * mainatain existing APIs for compatiblity with user provided code * fix compile issue * fix file writer integ test * throw exception instead of returning null * fix default handling, add comments * PR feedback * Make existing schema providers use deprecated methods to ensure users don't have failures on upgrade to 1.2.o * mark older method as deprecated and delegate to it for PostProcessor * fix schemaproviderwithpostprocessor * fix integ-test * fix integ test * make delegating schema provider impl final

the-other-tim-brown commented Nov 26, 2025

View reviewed changes

hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaProvider.java Outdated Show resolved Hide resolved

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Nov 26, 2025

the-other-tim-brown marked this pull request as ready for review December 1, 2025 15:03

the-other-tim-brown mentioned this pull request Dec 2, 2025

feat(schema): phase 12 - Perform Data Source Helpers Migration #14382

Merged

3 tasks

voonhous changed the title ~~feat(schema): Migrate SchemProviders in Hudi-Utilities to use HoodieSchema~~ feat(schema): Migrate SchemaProviders in Hudi-Utilities to use HoodieSchema Dec 3, 2025

voonhous reviewed Dec 3, 2025

View reviewed changes

the-other-tim-brown force-pushed the 14276-schema-providers branch from 142ce0e to 0ed42d1 Compare December 3, 2025 16:03

the-other-tim-brown added 13 commits December 3, 2025 11:04

start migrating schema provider classes

bea34a2

make code compile

edbf40b

fix test setup

be67d96

fix ExampleDataSchemaProvider

a26d95f

fix integ-test module

1b2ca52

handle null schema in deduce schema step

24385cf

fix getField and default value handling, fix test setups

5d7f743

mainatain existing APIs for compatiblity with user provided code

f76fdb3

fix compile issue

e395ac7

fix file writer integ test

3da78bf

throw exception instead of returning null

98ef683

fix default handling, add comments

d3e71f7

PR feedback

e92db40

the-other-tim-brown force-pushed the 14276-schema-providers branch from 0ed42d1 to e92db40 Compare December 3, 2025 16:04

bvaradar requested changes Dec 3, 2025

View reviewed changes

the-other-tim-brown added 6 commits December 3, 2025 22:16

Make existing schema providers use deprecated methods to ensure users…

85865f8

… don't have failures on upgrade to 1.2.o

mark older method as deprecated and delegate to it for PostProcessor

ecd86f8

fix schemaproviderwithpostprocessor

9a3385f

fix integ-test

90d5891

fix integ test

e88b045

make delegating schema provider impl final

7d81d49

bvaradar approved these changes Dec 4, 2025

View reviewed changes

the-other-tim-brown merged commit 49938cb into apache:master Dec 4, 2025
195 of 204 checks passed

the-other-tim-brown deleted the 14276-schema-providers branch December 4, 2025 18:16

Conversation

the-other-tim-brown commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bvaradar left a comment

Choose a reason for hiding this comment

Uh oh!

the-other-tim-brown commented Dec 4, 2025

Uh oh!

hudi-bot commented Dec 4, 2025

CI report:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

the-other-tim-brown commented Nov 26, 2025 •

edited

Loading