[fix](paimon) Read Paimon tables on BE without reconstructing the catalog metastore#65867
Open
morningman wants to merge 4 commits into
Open
[fix](paimon) Read Paimon tables on BE without reconstructing the catalog metastore#65867morningman wants to merge 4 commits into
morningman wants to merge 4 commits into
Conversation
… reads Paimon branch/time-travel/incremental/system-table reads over an HMS Paimon catalog fail on BE with: NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf paimon-hive-connector-3.1 marks hive-exec as `provided` and only bundles relocated hive classes under org.apache.paimon.shade.*, so the real org.apache.hadoop.hive.conf.HiveConf (needed by org.apache.paimon.hive.SerializableHiveConf when a HMS-backed Paimon table is deserialized on BE, and by HiveCatalog.createHiveConf) was never in paimon-scanner's own fat jar. It used to be provided accidentally by java-udf's hive-catalog-shade, which sits on BE's shared JVM classpath (bin/start_be.sh preloads both preload-extensions and java-udf, and that classloader is the parent of every per-scanner JniScannerClassLoader). apache#65733 replaced hive-catalog-shade with the slim hive-udf-shade, dropping HiveConf from the shared classpath and breaking paimon-scanner. Make paimon-scanner self-contained by bundling the Hive classes it actually needs: hive-common (HiveConf plus its construction closure) and hive-shims-common (shims/Utils, used by the lazy HiveConf#getUser). Both are pinned to ${hive.version} (3.1.x, matching the Hive the connector was built against and the previous hive-catalog-shade 3.1.1) and have their transitive trees pruned; hadoop is already on BE's shared classpath. Net jar growth is ~540KB (hive-common + hive-shims-common only), no jetty/orc/ant bloat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed. I found no substantiated blocking issue in the current one-file
change.
Part 1.3 checkpoint conclusions:
- Goal and proof: this restores the unrelocated
HiveConfand shims classes needed when FE
serializes an HMS-backed Paimon table and the BE Paimon JNI scanner deserializes it. Both
legacy and Format V2 affected reads converge on that boundary. The PR reports rebuilt-artifact
smoke coverage fornew HiveConf(), theSerializableHiveConfround trip, andgetUser.
Existing HMS regression coverage exercises forced JNI, incremental, time-travel, and
system-table modes; the named branch product test was not run here. - Scope and clarity: this is a focused one-file dependency repair. Wildcard transitive pruning
keeps the addition to the two small Hive artifacts rather than restoring the former catalog
shade. - Concurrency and lifecycle: no threading, locking, mutable-state ownership, or resource
lifecycle changes. The existing process-lifetime scanner classloader behavior is unchanged. - Configuration and compatibility: no configuration, protocol, storage-format, persistence, or
serialized-wrapper contract changes. The Hive versions remain within the existing 3.1 family. - Parallel paths: legacy and Format V2 JNI modes use the same Paimon scanner artifact; native
and C++ reader paths do not cross this Java deserialization boundary, and other scanners are
isolated. - Tests and results: no result files changed. Visible CI style, license, and dependency checks
pass, but they did not execute the deployed Paimon fat jar. The review instructions prohibited
local builds/tests, so validation here is static plus the PR's reported manual artifact tests. - Observability, transactions, writes, FE-BE variables, and performance: no logging/metric,
transaction, data-write, or FE-BE variable behavior changes. The only identified cost is the
modest jar/startup footprint; no query hot path changes. - Risk closure: the wildcard-pruned external dependency exercised by
HiveConfis already
supplied by the supported BE parent classpath, and the direct Hive jars follow the existing
assembly, deployment, and parent-first loading path. I found no substantiated closure,
collision, or version issue.
User focus: no additional focus was provided.
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
TPC-H: Total hot run time: 29445 ms |
Contributor
TPC-DS: Total hot run time: 176734 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
…n HMS reads Reading an HMS-backed Paimon table (branch/time-travel/incremental/system tables) over an HMS Paimon catalog fails on BE with: NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException This is the metastore-API counterpart of the HiveConf gap fixed in the previous commit. When BE deserializes an HMS-backed Paimon table, the serialized graph's org.apache.paimon.hive.HiveCatalogLoader links org.apache.paimon.hive.HiveCatalog, which references a broad slice of org.apache.hadoop.hive.metastore.api.* (Table, Database, Partition, StorageDescriptor, SerDeInfo, FieldSchema, NoSuchObjectException, UnknownTableException, ...). paimon-hive-connector-3.1 marks hive-exec `provided` and only bundles relocated hive under org.apache.paimon.shade.*, so this real `api` package was never in paimon-scanner's own fat jar. It used to be provided accidentally by java-udf's hive-catalog-shade on BE's shared classpath, but apache#65733 replaced that with the slim hive-udf-shade. In Hive 3.x the whole thrift-generated metastore `api` package lives in hive-standalone-metastore (the `hive-metastore` artifact is a thin shim and does NOT contain it). Bundle it, pinned to ${hive.version} (3.1.x) like the two hive jars added previously, with its transitive tree pruned: the api classes only extend org.apache.thrift.* (already on BE's shared classpath via java-common's libthrift 0.16, exactly as under the old hive-catalog-shade, which itself bundled no plain thrift) and the java stdlib; datanucleus/orc/hadoop/jetty/etc. are only needed by the metastore server/client, which BE never runs (it reads the already-resolved table). Net add is the ~11MB metastore api package only, no jetty/orc/datanucleus bloat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHwqsiGWNosZCeS4ecK97e
…alog metastore Reading a Paimon system table (e.g. $snapshots) over a metastore-backed catalog (HMS / DLF) failed on BE with: ClassNotFoundException: com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient SnapshotManager#latestSnapshotId resolves the latest snapshot through the table's catalog SnapshotLoader, which on BE reconstructs the catalog's metastore client and its whole Hive/DLF REST stack (okhttp, tea, hive-exec's SessionState, ...) — even though the BE only reads and the snapshots already live on the object store. Serialize a catalog-less table to the BE for every Paimon read (PaimonScanNode): - data table: rebuild via FileStoreTableFactory with an empty CatalogEnvironment. fileIO/location/schema are preserved, and dynamic options (time travel / incremental) are already merged into the schema by copy(), so nothing is lost. - system table: rebuild it over such a catalog-less data table via SystemTableLoader. With no catalog loader, SnapshotManager#latestSnapshotId falls back to listing the snapshot directory on the filesystem, and the BE never deserializes a HiveCatalogLoader. paimon-scanner therefore no longer needs any Hive classes on the BE, so the hive-common / hive-shims-common / hive-standalone-metastore jars are removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29714 ms |
Contributor
TPC-DS: Total hot run time: 176867 ms |
Contributor
ClickBench: Total hot run time: 24.88 s |
… BE reads Reading an object-store-backed Paimon table over a REST/DLF catalog failed on BE when resolving the latest snapshot from the filesystem: NoClassDefFoundError: org/apache/paimon/fs/FileIOLoader The paimon OSS/S3 FileIO plugins (paimon-s3 -> S3Loader, paimon-jindo -> JindoLoader) were bundled in preload-extensions, which sits on BE's JVM system (app) classpath, but the org.apache.paimon.fs.FileIOLoader interface they implement lives in paimon-common, bundled only in paimon-scanner's own JniScannerClassLoader. When a BE read resolves the latest snapshot from the filesystem, RESTTokenFileIO.fileIO() calls FileIO.get() -> ServiceLoader.load(FileIOLoader.class, ...), which eagerly instantiates every registered provider. JniScannerClassLoader is a plain parent-first URLClassLoader, so the app classloader defines S3Loader/JindoLoader and cannot resolve the child-only FileIOLoader interface -> the whole discovery aborts. Co-locate the plugins with the FileIOLoader interface: move paimon-s3 / paimon-jindo from preload-extensions to paimon-scanner. paimon-scanner already carries a complete paimon runtime, and the assembly's metaInf-services handler merges the plugins' service files with paimon-common's, so all five FileIOLoader providers (local/hadoop/viewfs/s3/jindo) and the interface end up in one classloader. The Jindo SDK stays on the app classpath (start_be.sh adds jindofs to DORIS_CLASSPATH), so paimon-scanner still reaches it via parent delegation. Both jars must be rebuilt and redeployed: paimon-scanner (now carries the plugins) and preload-extensions (must stop advertising the un-linkable providers on the shared classpath). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHwqsiGWNosZCeS4ecK97e
morningman
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Reading a Paimon table over a metastore-backed or REST catalog (HMS / DLF) fails on
BE whenever the JNI reader is used — most visibly system tables (e.g.
$snapshots,$files), which always go through JNI, and also branch / time-travel / incrementalreads. Two failures were observed, both caused by the BE rebuilding a catalog it does
not need:
Root cause. A table loaded from a metastore-backed Paimon catalog carries a Paimon
CatalogLoader(e.g.HiveCatalogLoader) in itsCatalogEnvironment. When FEserializes that table to BE,
SnapshotManager#latestSnapshotIdresolves the latestsnapshot through the catalog's
SnapshotLoader, which on BE reconstructs the catalog'smetastore client and its whole Hive / DLF-REST stack — even though the BE only reads
(via FE-resolved splits and the object store) and the snapshots already live there.
This was previously masked by
java-udf's ~122MBhive-catalog-shadeon BE's sharedclasspath; #65733 replaced it with the slim
hive-udf-shade, exposing the dependency.Fix 1 — serialize a catalog-less table to the BE (
PaimonScanNode).FileStoreTableFactorywith an emptyCatalogEnvironment.A
FileStoreTableis fully defined by fileIO / location / schema, and its dynamicoptions (time travel, incremental) are already merged into the schema by
copy(...),so nothing is lost except the catalog loader.
SystemTableLoader.With no catalog loader,
SnapshotManager#latestSnapshotIdlists the snapshot directoryon the filesystem instead of calling the metastore, so the BE never reconstructs the
catalog and no longer needs any Hive / metastore classes.
Fix 2 — co-locate the Paimon FileIO plugins with
paimon-scanner(packaging).Reading the snapshot from the filesystem makes the BE materialize the object-store
FileIOlazily viaFileIO.get()→ServiceLoader.load(FileIOLoader.class, ...),which eagerly instantiates every registered provider. The OSS/S3 plugins
(
paimon-s3→S3Loader,paimon-jindo→JindoLoader) were bundled inpreload-extensions, on BE's JVM system (app) classpath, but theorg.apache.paimon.fs.FileIOLoaderinterface they implement ships inpaimon-common,bundled only in paimon-scanner's own
JniScannerClassLoader. That loader isparent-first, so the app classloader defines
S3Loader/JindoLoaderand cannotresolve the child-only
FileIOLoader→NoClassDefFoundError: FileIOLoader, whichaborts discovery. Moving the two plugins into
paimon-scannerputs the whole FileIOSPI (interface + all providers) in one classloader; the Jindo SDK stays on the app
classpath (
start_be.shaddsjindofstoDORIS_CLASSPATH) and is still reachablevia parent delegation.
Regression tests to re-run:
io.trino.tests.product.paimon.TestPaimonSparkCompatibility(system-table reads) andexternal_table_p2/paimon/test_paimon_dlf_rest_catalog.Release note
Fix Paimon reads (system tables, branch / time-travel / incremental) failing on BE over
metastore-backed or REST catalogs (HMS / DLF) with
NoClassDefFoundError(
HiveConf/FileIOLoader) orClassNotFoundException: ProxyMetaStoreClient.Check List (For Author)
Test
Behavior changed:
Does this need documentation?