Skip to content

[core] Support source table FileIO for BLOB descriptors - #8693

Merged
JingsongLi merged 3 commits into
apache:masterfrom
LsomeYeah:codex/blob-descriptor-source-table
Jul 21, 2026
Merged

[core] Support source table FileIO for BLOB descriptors#8693
JingsongLi merged 3 commits into
apache:masterfrom
LsomeYeah:codex/blob-descriptor-source-table

Conversation

@LsomeYeah

@LsomeYeah LsomeYeah commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

When a source table exposes BLOB values as descriptors, a downstream write must reopen the referenced objects before writing them into the target table. Rebuilding FileIO from the target table catalog context can miss source data permissions, especially for token-based catalogs.

This PR adds the optional blob-descriptor.source-table option. When configured, Paimon loads that table from the same catalog and uses its serializable FileIO to materialize descriptor-backed BLOBs. The existing target-context behavior remains unchanged when the option is absent.

Changes

  • Resolve the configured source table once and use its FileIO for non-HTTP descriptor URIs.
  • Pre-initialize lazy source credentials before distributing the reader factory, avoiding per-task REST token requests.
  • Resolve the reader factory outside distributed Spark tasks, including postpone compaction.
  • Preserve the existing public Flink mapToInternalRow API and route the new factory path through a private helper.
  • Fail fast when blob-descriptor.source-table is used by a target without a catalog loader, including REST external tables.
  • Document same-catalog and branch usage, option precedence, and the unsupported REST external target case.

Compatibility and limitations

  • The source table must belong to the same catalog; a branch suffix is supported.
  • REST external target tables are not supported because they do not provide a catalog loader.
  • blob-descriptor.source-table takes precedence over other blob-descriptor.* FileIO options.
  • Without this option, the existing target CatalogContext behavior is unchanged.

Tests

  • BlobDescriptorReaderFactoryTest and UriReaderFactoryTest
  • Flink SQL end-to-end BLOB materialization in BlobTableITCase
  • Spark V1 and V2 SQL end-to-end BLOB materialization in BlobTestBase
  • Spark V2 writer-factory regression test in PaimonBatchWriteTest
  • Spark postpone compaction test verifying that the source table is not resolved in tasks
  • Flink null-overload source compatibility and row-wrapper compatibility tests
  • Configuration documentation completeness test
  • Spark 4 compilation with JDK 17
  • Flink 2 compilation with JDK 11

@LsomeYeah
LsomeYeah force-pushed the codex/blob-descriptor-source-table branch from 2ea960b to 0ae8567 Compare July 16, 2026 09:05
@steFaiz

steFaiz commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Thanks for this PR!
Does the target table store the blobs from source table as blob format or blob descriptors?
If as blob format, I think we can use blob-descriptor.XXX to specify options for CatalogContext as in #7231
If as blob descriptors, I think blob-view is designed to solve this.

private static final long serialVersionUID = -8477284718943635074L;

@Nullable private final CatalogContext context;
@Nullable private final FileIO fileIO;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1、FileIO should be transient
2、It seems strange to hold both fileIO and readers

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just overwrite newReader() to just put your special fileIO in

Set<Integer> blobFields) {
this.row = row;
this.uriReaderFactory = new UriReaderFactory(catalogContext);
this.uriReaderFactory = uriReaderFactory;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If FlinkRowWrapper with uriReaderFactory(fileIO), then it can't be successfully serialize and deserialize. FileIO is not serializable.

@LsomeYeah

Copy link
Copy Markdown
Contributor Author

Thanks for this PR! Does the target table store the blobs from source table as blob format or blob descriptors? If as blob format, I think we can use blob-descriptor.XXX to specify options for CatalogContext as in #7231 If as blob descriptors, I think blob-view is designed to solve this.

@steFaiz
Thanks for the suggestion!
The target stores an independent managed BLOB copy: it reads the blob through the source descriptor and writes it into the target’s own BLOB storage.
blob-descriptor.* only provides static CatalogContext options, while the source table’s data token is dynamically refreshed by its FileIO and cannot be reliably materialized into an options map. blob-view is a no-copy reference with different semantics.
Hope this clarifies the motivation for source-table.

@steFaiz

steFaiz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@LsomeYeah Thanks! Make sense to me!

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 162fae3 into apache:master Jul 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants