[core] Support configurable Iceberg metadata path independent of storage type

[jahangir-yelp]

[core] Support configurable Iceberg metadata path independent of storage type

Purpose

Linked issue [Feature]: close #5470

This PR introduces a new table option metadata.iceberg.storage-location to allow users to configure where Iceberg metadata is stored, independently of the selected metadata.iceberg.storage type.

By default, when using metadata.iceberg.storage = table-location, Paimon does not register the table or database in Hive or AWS Glue catalogs. This is expected behavior, but it limits integration with catalog-based systems such as Lake Formation or Iceberg readers that rely on Hive/Glue.

This feature adds support for catalog registration even when using table-location, by decoupling the metadata storage path from the catalog integration logic.

This enables users to:

• Store Iceberg metadata in the table directory
• Still register tables and databases in Hive or Glue
• Support hybrid deployments where metadata is colocated with table data but still discoverable via catalogs

Tests

• Added unit tests in IcebergCommitCallbackTest
• Manually tested using Kafka to Paimon connector with Flink and AWS Glue

API and Format

No changes to public API or storage format

Documentation

Introduces a new table option: metadata.iceberg.storage-location
Updated iceberg-compatibility.md to document the new configuration and usage scenarios

[JingsongLi]

+1

[JingsongLi]

Thanks @jahangir-yelp for the contribution!