[core] Support source table FileIO for BLOB descriptors - #8693
Conversation
2ea960b to
0ae8567
Compare
|
Thanks for this PR! |
| private static final long serialVersionUID = -8477284718943635074L; | ||
|
|
||
| @Nullable private final CatalogContext context; | ||
| @Nullable private final FileIO fileIO; |
There was a problem hiding this comment.
1、FileIO should be transient
2、It seems strange to hold both fileIO and readers
There was a problem hiding this comment.
You can just overwrite newReader() to just put your special fileIO in
| Set<Integer> blobFields) { | ||
| this.row = row; | ||
| this.uriReaderFactory = new UriReaderFactory(catalogContext); | ||
| this.uriReaderFactory = uriReaderFactory; |
There was a problem hiding this comment.
If FlinkRowWrapper with uriReaderFactory(fileIO), then it can't be successfully serialize and deserialize. FileIO is not serializable.
@steFaiz |
|
@LsomeYeah Thanks! Make sense to me! |
|
+1 |
Purpose
When a source table exposes BLOB values as descriptors, a downstream write must reopen the referenced objects before writing them into the target table. Rebuilding
FileIOfrom the target table catalog context can miss source data permissions, especially for token-based catalogs.This PR adds the optional
blob-descriptor.source-tableoption. When configured, Paimon loads that table from the same catalog and uses its serializableFileIOto materialize descriptor-backed BLOBs. The existing target-context behavior remains unchanged when the option is absent.Changes
FileIOfor non-HTTP descriptor URIs.mapToInternalRowAPI and route the new factory path through a private helper.blob-descriptor.source-tableis used by a target without a catalog loader, including REST external tables.Compatibility and limitations
blob-descriptor.source-tabletakes precedence over otherblob-descriptor.*FileIO options.CatalogContextbehavior is unchanged.Tests
BlobDescriptorReaderFactoryTestandUriReaderFactoryTestBlobTableITCaseBlobTestBasePaimonBatchWriteTest