[api][core][spark] Support custom partition locations

[sundapeng]

Summary

Support Spark ADD PARTITION ... LOCATION for internal, catalog-managed Format Tables. Custom locations are stored in the generic partition options map under path.

Changes

• Extend the existing POST .../partitions request with position-aligned partitionOptions. The endpoint and response stay unchanged, and unknown options are preserved.
• Resolve and validate options["path"] in Core. Custom locations must be absolute, outside the table directory, and non-overlapping.
• Read registered custom locations while preventing Paimon from writing, deleting, truncating, or analyzing their data.
• Map Spark partition location to Paimon path. DROP PARTITION only unregisters a custom location, and MSCK REPAIR TABLE leaves it unchanged.

Testing

• REST API JSON, REST catalog, caching catalog, and delegate catalog tests
• Core Format Table partition manager, scan, commit, and statistics suites: 174 tests
• Spark partition management, DDL, ANALYZE, MSCK, and repair suites: 129 tests
• REST OpenAPI validation: 56 catalog operations and 6 management operations
• Spotless, Checkstyle, and diff checks

Notes

Upgrade the REST server and every reader before registering custom locations. PyPaimon does not support them yet and will be handled separately.

The commits are ordered REST API → Core → Spark.

Catalog implementations overriding the extended createPartitions method must accept the new partitionOptions argument.

Follow-up to #8750 and #8751.

[JingsongLi]

Requirement fit: SUPPORTED
Implementation: FINDINGS

[JingsongLi]

[P1] Validate the prospective registry atomically before registering locations.

This path validates each explicit location only against the table root and the default directory of that same spec, then immediately mutates the catalog. A batch can therefore assign the same or nested LOCATION values to different specs; MSCK can similarly add a default spec whose directory is already owned by another explicit spec. Registration succeeds, but scans, writes, and DROP later reject the full registry as overlapping, so normal SQL cannot even unregister the bad entry. Please make the catalog/REST mutation atomically validate the complete post-update ownership set for both ordinary create and /with-locations; a client-side listing alone would still race concurrent ADDs.

[JingsongLi]

[P1] Canonicalize the filesystem identity before comparing ownership.

The ownership key uses the raw lower-cased scheme and authority, although different URI spellings can resolve to the same filesystem. For example, Hadoop can resolve hdfs://nn and hdfs://nn:8020 to the same NameNode, but this code treats them as separate roots. A table at hdfs://nn:8020/warehouse/t can therefore accept an explicit location such as hdfs://nn/warehouse/t, bypassing the table-root check and potentially reading the whole table as one partition or writing through a default sibling into explicitly owned data. Please derive the key from the resolved/canonical FileSystem URI, including default ports and configured aliases, and use that identity consistently for overlap checks and FileIO routing.

[JingsongLi]

[P1] Do not reject URI forms used by supported filesystems.

These checks reject hdfs:///path because it has no authority, even though Paimon documents that form and HadoopFileIO can resolve it through the catalog Hadoop configuration. They also reject the standard ABFS form abfs://filesystem@account.dfs.core.windows.net/path, because java.net.URI parses filesystem as user info even though it is the Azure filesystem/container name rather than a credential. As a result, explicit partition locations are unusable for common HDFS deployments and standard ADLS Gen2 addresses. Please make validation scheme/FileIO-aware: qualify authorityless HDFS through CatalogContext, recognize the structural ABFS authority, and continue rejecting actual embedded credentials.

[JingsongLi]

[P1] Preserve the documented unknown-count fallback.

The OpenAPI description explicitly allows explicitPartitionLocationCount to be omitted, and Catalog#getExplicitPartitionLocationCount defines an empty result as a signal to load and validate the full registry. Requiring the count whenever the server advertises the location capability contradicts both contracts: a location-capable provider that cannot supply this optional optimization will make every getTable and table-detail conversion fail before any fallback can run. Please accept null as OptionalLong.empty() and use the existing full-registry path, or advertise count support through a separate capability that makes the field mandatory.

[JingsongLi]

[P2] Validate the full registry before scoped ANALYZE.

This query is already prefix-pruned, so checking location only on the returned partitions does not validate global path ownership or the authoritative explicit-location count. If selected default partition A has a directory that is also owned by an unselected explicit partition B, this guard passes and FormatTablePartitionStatsCollector reads that directory as A, probing explicitly owned data and replacing A statistics with B data. That state is reachable through the current ADD/MSCK registration gap or an inconsistent provider. Please run the same count-aware full-registry preflight used by scans and commits, then apply the ANALYZE prefix and reject selected explicit partitions.

[JingsongLi]

Incremental review of the latest revision.

[JingsongLi]

[P2] Avoid scanning the complete registry before every normal read. When no custom location exists, this loop must drain every page, and FormatTableScan.plan() invokes it unconditionally for every catalog-managed Format Table read (the first write does the same). A table with one million default-location partitions adds roughly 1,000 serial REST requests before planning can begin. Please expose an authoritative server-side has/count signal or negotiate a capability that lets PyPaimon reject unsupported custom-location tables without enumerating the entire registry.

[JingsongLi]

I think location should be options, just like Table. Create partitions should also support options instead of location.

[JingsongLi]

More importantly, it is a highly specific variable—not a general-purpose one—so it should be tucked into the options object.

[JingsongLi]

+1