Skip to content

[core] Support custom table paths in JdbcCatalog#7475

Draft
tub wants to merge 3 commits into
apache:masterfrom
tub:jdbc-custom-table-path
Draft

[core] Support custom table paths in JdbcCatalog#7475
tub wants to merge 3 commits into
apache:masterfrom
tub:jdbc-custom-table-path

Conversation

@tub
Copy link
Copy Markdown
Contributor

@tub tub commented Mar 18, 2026

Summary

  • Enable users to specify custom storage locations for individual tables in JdbcCatalog via CoreOptions.PATH, replicating the pattern already used by HiveCatalog
  • The catalog-level warehouse becomes a default, but individual tables can live at arbitrary paths persisted in paimon_table_properties
  • Custom-path tables behave as external tables: drop skips filesystem deletion, rename skips filesystem move

Test plan

  • testCreateTableWithCustomPath — schema at custom location, getTableLocation() returns it, path stored in JDBC
  • testCreateTableWithCustomPathSyncDisabledpath stored even when syncTableProperties=false
  • testDropTableWithCustomPath — JDBC cleaned up, data NOT deleted
  • testDropTableWithDefaultPath — data IS deleted (existing behavior preserved)
  • testRenameTableWithCustomPath — metadata updated, path preserved, no filesystem rename
  • testAlterTableWithCustomPath — path preserved after alter
  • testLoadTableSchemaWithCustomPath — schema loads from custom location
  • testGetTableLocationFallbackgetTableLocation() returns default when no stored path
  • All 59 JdbcCatalogTest tests pass (including inherited CatalogTestBase tests)

🤖 Generated with Claude Code

tub and others added 2 commits March 18, 2026 10:50
Enable users to specify custom storage locations for individual tables
via CoreOptions.PATH, replicating the pattern already used by HiveCatalog.
The catalog-level warehouse becomes a default, but individual tables can
live at arbitrary paths persisted in paimon_table_properties.

Key changes:
- JdbcUtils: add getTableProperty() to fetch a single table property
- JdbcCatalog: override allowCustomTablePath(), getTableLocation()
- createTableImpl: use initialTableLocation(), store path for custom tables
- dropTableImpl: skip filesystem deletion for custom-path tables
- renameTableImpl: skip filesystem rename for custom-path tables
- alterTableImpl: preserve custom path across property refresh
- dropDatabaseImpl/repairTable: always manage table properties

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Custom table path support is now only available when
syncTableProperties is enabled, preserving the existing default
behavior when sync is off.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub
Copy link
Copy Markdown
Contributor Author

tub commented Mar 18, 2026

@nickdelnano I built this on top of your table properties change. It'll allow folks to have a fully fledged catalog spanning multiple warehouses/buckets etc.

- Consolidate getTableLocation + isCustomTablePath into a single
  fetchStoredPathIfSyncEnabled call in dropTableImpl and renameTableImpl
- Document that repairTable cannot recover custom-path tables when
  all JDBC metadata is lost
- Fix stale "Hive Metastore" comment in renameTableImpl

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-documented PR with comprehensive test plan. Supporting custom table paths in JdbcCatalog aligns with HiveCatalog behavior.

Review:

  1. External table semantics: Custom-path tables treated as external (drop skips filesystem deletion, rename skips filesystem move) — correct and consistent with Hive conventions.

  2. Path persistence: Stored in paimon_table_properties — the existing JDBC metadata table. No schema migration needed.

  3. Fallback: getTableLocation() returns default warehouse path when no custom path stored. Backward compatible.

  4. Test coverage: 9 specific tests plus all 59 inherited CatalogTestBase tests passing — excellent.

  5. 388 additions across JdbcCatalog, JdbcUtils, and tests. Reasonable for the scope.

  6. Edge cases to verify:

    • Custom path points to a non-existent location → create table should fail or create directory?
    • Custom path is on a different filesystem than warehouse → does FileIO resolve correctly?
    • Two tables pointing to the same custom path → should be rejected?

LGTM. Clean implementation following established patterns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants