[core] Support custom table paths in JdbcCatalog#7475
Conversation
Enable users to specify custom storage locations for individual tables via CoreOptions.PATH, replicating the pattern already used by HiveCatalog. The catalog-level warehouse becomes a default, but individual tables can live at arbitrary paths persisted in paimon_table_properties. Key changes: - JdbcUtils: add getTableProperty() to fetch a single table property - JdbcCatalog: override allowCustomTablePath(), getTableLocation() - createTableImpl: use initialTableLocation(), store path for custom tables - dropTableImpl: skip filesystem deletion for custom-path tables - renameTableImpl: skip filesystem rename for custom-path tables - alterTableImpl: preserve custom path across property refresh - dropDatabaseImpl/repairTable: always manage table properties Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Custom table path support is now only available when syncTableProperties is enabled, preserving the existing default behavior when sync is off. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@nickdelnano I built this on top of your table properties change. It'll allow folks to have a fully fledged catalog spanning multiple warehouses/buckets etc. |
- Consolidate getTableLocation + isCustomTablePath into a single fetchStoredPathIfSyncEnabled call in dropTableImpl and renameTableImpl - Document that repairTable cannot recover custom-path tables when all JDBC metadata is lost - Fix stale "Hive Metastore" comment in renameTableImpl Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
JingsongLi
left a comment
There was a problem hiding this comment.
Well-documented PR with comprehensive test plan. Supporting custom table paths in JdbcCatalog aligns with HiveCatalog behavior.
Review:
-
External table semantics: Custom-path tables treated as external (drop skips filesystem deletion, rename skips filesystem move) — correct and consistent with Hive conventions.
-
Path persistence: Stored in
paimon_table_properties— the existing JDBC metadata table. No schema migration needed. -
Fallback:
getTableLocation()returns default warehouse path when no custom path stored. Backward compatible. -
Test coverage: 9 specific tests plus all 59 inherited
CatalogTestBasetests passing — excellent. -
388 additions across JdbcCatalog, JdbcUtils, and tests. Reasonable for the scope.
-
Edge cases to verify:
- Custom path points to a non-existent location → create table should fail or create directory?
- Custom path is on a different filesystem than warehouse → does FileIO resolve correctly?
- Two tables pointing to the same custom path → should be rejected?
LGTM. Clean implementation following established patterns.
Summary
CoreOptions.PATH, replicating the pattern already used by HiveCatalogwarehousebecomes a default, but individual tables can live at arbitrary paths persisted inpaimon_table_propertiesTest plan
testCreateTableWithCustomPath— schema at custom location,getTableLocation()returns it,pathstored in JDBCtestCreateTableWithCustomPathSyncDisabled—pathstored even whensyncTableProperties=falsetestDropTableWithCustomPath— JDBC cleaned up, data NOT deletedtestDropTableWithDefaultPath— data IS deleted (existing behavior preserved)testRenameTableWithCustomPath— metadata updated, path preserved, no filesystem renametestAlterTableWithCustomPath— path preserved after altertestLoadTableSchemaWithCustomPath— schema loads from custom locationtestGetTableLocationFallback—getTableLocation()returns default when no stored path🤖 Generated with Claude Code