Skip to content

fix(client): fix NPE in schema conflict resolution on commits with null writer schema - #19388

Merged
voonhous merged 2 commits into
apache:masterfrom
yihua:fix-schema-conflict-npe-inflight-instant
Jul 29, 2026
Merged

fix(client): fix NPE in schema conflict resolution on commits with null writer schema#19388
voonhous merged 2 commits into
apache:masterfrom
yihua:fix-schema-conflict-npe-inflight-instant

Conversation

@yihua

@yihua yihua commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Describe the issue this Pull Request addresses

When schema conflict resolution is enabled and a writer commits a batch whose writer schema is the Avro null schema (for example, an ingestion round that writes no data), SimpleSchemaConflictResolutionStrategy.resolveConcurrentSchemaEvolution resolves the table schema at the current transaction's owner instant. At pre-commit time that instant is inflight and has no completion time, so the completion-time filter in ConcurrentSchemaEvolutionTableSchemaGetter.getLastCommitMetadataWithValidSchemaFromTimeline calls String.compareTo(null), and the commit fails:

Caused by: org.apache.hudi.exception.HoodieException: Unable to get table schema
	at org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy.getTableSchemaAtInstant(SimpleSchemaConflictResolutionStrategy.java:173)
	at org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy.resolveConcurrentSchemaEvolution(SimpleSchemaConflictResolutionStrategy.java:74)
	...
Caused by: java.lang.NullPointerException
	at java.lang.String.compareTo(String.java:1155)
	at org.apache.hudi.common.table.timeline.InstantComparison.lambda$static$3(InstantComparison.java:34)
	at org.apache.hudi.common.table.timeline.InstantComparison.compareTimestamps(InstantComparison.java:38)
	at org.apache.hudi.client.transaction.ConcurrentSchemaEvolutionTableSchemaGetter.lambda$getLastCommitMetadataWithValidSchemaFromTimeline$3(ConcurrentSchemaEvolutionTableSchemaGetter.java:175)

The defect is twofold, and Hudi 1.x writes both table versions:

  • On table version 6, ordering the schema evolution timeline by completion time is wrong to begin with: version 6 does not record completion time on disk (the in-memory value is synthesized from the meta file modification time), and 0.x writers concurrently writing the same table order the timeline by requested time. Requested-time ordering restores 0.x parity and also removes the NPE on this version, since every instant carries a requested time.
  • On table version 8 and above, completion-time ordering is correct, but the same NPE fires for an empty-batch commit: the current transaction's owner instant is not completed yet and has no completion time.

A transaction with a null writer schema does not evolve the schema, so the resolution should adopt the current table schema.

Summary and Changelog

  • The per-timeline-version instant ordering is exposed as a first-class API on InstantComparator: orderingComparator() / getOrderingTime(instant) (requested-time based in v1, completion-time based in v2), so version handling lives in the timeline layer rather than in callers.
  • ConcurrentSchemaEvolutionTableSchemaGetter: sorts and bounds the schema evolution timeline with that ordering (completion time for table version 8 and above, requested time for earlier versions, matching 0.x). A target instant without an ordering time (not completed yet, on table version 8 and above) does not bound the lookup instead of throwing NPE.
  • SimpleSchemaConflictResolutionStrategy: the null-writer-schema path adopts the table schema as of the current transaction's owner instant. On table version 6 the lookup is bounded by the instant's requested time (matching 0.x behavior); on table version 8 and above the inflight instant carries no completion time, so the latest table schema is resolved. The prior-instant lookup for the transaction start snapshot also uses the version-appropriate ordering time.
  • Tests: TestSimpleSchemaConflictResolutionStrategy gains the inflight-instant NPE regression plus table-version-6 cases proving requested-time bounding; TestConcurrentSchemaEvolutionTableSchemaGetter gains latest-schema ordering regressions with the same two-commit layout on both versions (the earlier-requested commit completes last), so table version 8 and above must return the completion-time winner and table version 6 must return the requested-time winner even when the synthesized completion order disagrees; TestInstantComparators covers the new InstantComparator ordering APIs directly.

Impact

Fixes commit failures on multi-writer tables for ingestion rounds that write no data, on table version 6 and table version 8 and above alike. No public API or user-facing behavior change otherwise.

Risk Level

low

The new regression tests fail with the production NPE signature before the fix and pass after; the table-version-6 ordering tests fail without the version-aware ordering and pass with it; both full test classes pass.

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

…ll writer schema

When schema conflict resolution is enabled and a writer commits a batch whose
writer schema is the Avro null schema (an ingestion round that writes no data),
the resolution strategy resolves the table schema at the current transaction's
owner instant. At pre-commit time that instant is inflight with no completion
time, so the completion-time filter in the schema getter calls
String.compareTo(null) and the commit fails with an NPE.

- Expose the per-timeline-version instant ordering as a first-class API on
  InstantComparator: orderingComparator() / getOrderingTime(instant),
  requested-time based in v1 and completion-time based in v2.
- ConcurrentSchemaEvolutionTableSchemaGetter sorts and bounds the schema
  evolution timeline with that ordering (completion time for table version 8
  and above, requested time for earlier versions, matching 0.x). A target
  instant without an ordering time no longer bounds the lookup instead of
  throwing.
- SimpleSchemaConflictResolutionStrategy adopts the table schema as of the
  owner instant on the null-writer-schema path, using the version-appropriate
  ordering time.
- Add regression coverage in TestSimpleSchemaConflictResolutionStrategy,
  TestConcurrentSchemaEvolutionTableSchemaGetter, and TestInstantComparators.
@yihua
yihua marked this pull request as ready for review July 28, 2026 05:58

@hudi-agent hudi-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! The PR fixes an NPE in schema conflict resolution when a writer commits a null (empty) writer schema, by introducing a timeline-version-aware "ordering time" (completion time for layout v2, requested time for v1) and using it in place of hardcoded completion time. I traced the write/pre-commit path across both table versions, the caller in TransactionUtils, and the InstantComparator interface change: the inflight-target NPE is properly guarded via the isNullOrEmpty(getOrderingTime(...)) short-circuit, the v1/v2 filter-and-comparator pairs stay consistent, the interface addition has no external implementors, and the blast radius is contained to the conflict-resolution path. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. A couple of small naming and clarity suggestions below.

cc @yihua

@github-actions github-actions Bot added the size:M PR with lines of changes in (100, 300] label Jul 28, 2026
…ion 6 and 8; docs(common): ordering-key invariant and upgrade-boundary completion time
@github-actions github-actions Bot added size:L PR with lines of changes in (300, 1000] and removed size:M PR with lines of changes in (100, 300] labels Jul 28, 2026
@yihua

yihua commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Comments are addressed.

@hudi-bot

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@voonhous voonhous left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All my comments addressed.
LGTM

@voonhous
voonhous merged commit b475651 into apache:master Jul 29, 2026
143 of 144 checks passed
voonhous pushed a commit that referenced this pull request Aug 6, 2026
…ll writer schema (#19388)

* fix(client): fix NPE in schema conflict resolution on commits with null writer schema

When schema conflict resolution is enabled and a writer commits a batch whose
writer schema is the Avro null schema (an ingestion round that writes no data),
the resolution strategy resolves the table schema at the current transaction's
owner instant. At pre-commit time that instant is inflight with no completion
time, so the completion-time filter in the schema getter calls
String.compareTo(null) and the commit fails with an NPE.

- Expose the per-timeline-version instant ordering as a first-class API on
  InstantComparator: orderingComparator() / getOrderingTime(instant),
  requested-time based in v1 and completion-time based in v2.
- ConcurrentSchemaEvolutionTableSchemaGetter sorts and bounds the schema
  evolution timeline with that ordering (completion time for table version 8
  and above, requested time for earlier versions, matching 0.x). A target
  instant without an ordering time no longer bounds the lookup instead of
  throwing.
- SimpleSchemaConflictResolutionStrategy adopts the table schema as of the
  owner instant on the null-writer-schema path, using the version-appropriate
  ordering time.
- Add regression coverage in TestSimpleSchemaConflictResolutionStrategy,
  TestConcurrentSchemaEvolutionTableSchemaGetter, and TestInstantComparators.

* test(client): parameterize schema conflict resolution over table version 6 and 8; docs(common): ordering-key invariant and upgrade-boundary completion time

(cherry picked from commit b475651)
voonhous pushed a commit that referenced this pull request Aug 7, 2026
…ll writer schema (#19388)

* fix(client): fix NPE in schema conflict resolution on commits with null writer schema

When schema conflict resolution is enabled and a writer commits a batch whose
writer schema is the Avro null schema (an ingestion round that writes no data),
the resolution strategy resolves the table schema at the current transaction's
owner instant. At pre-commit time that instant is inflight with no completion
time, so the completion-time filter in the schema getter calls
String.compareTo(null) and the commit fails with an NPE.

- Expose the per-timeline-version instant ordering as a first-class API on
  InstantComparator: orderingComparator() / getOrderingTime(instant),
  requested-time based in v1 and completion-time based in v2.
- ConcurrentSchemaEvolutionTableSchemaGetter sorts and bounds the schema
  evolution timeline with that ordering (completion time for table version 8
  and above, requested time for earlier versions, matching 0.x). A target
  instant without an ordering time no longer bounds the lookup instead of
  throwing.
- SimpleSchemaConflictResolutionStrategy adopts the table schema as of the
  owner instant on the null-writer-schema path, using the version-appropriate
  ordering time.
- Add regression coverage in TestSimpleSchemaConflictResolutionStrategy,
  TestConcurrentSchemaEvolutionTableSchemaGetter, and TestInstantComparators.

* test(client): parameterize schema conflict resolution over table version 6 and 8; docs(common): ordering-key invariant and upgrade-boundary completion time

(cherry picked from commit b475651)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L PR with lines of changes in (300, 1000]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants