[#11951] feat(iceberg): map V3 unknown type to Gravitino NullType

[nevzheng]

What changes were proposed in this pull request?

Map Iceberg V3's unknown type to Gravitino's existing Types.NullType, so an unknown column loads through the native metadata API as a first-class null type instead of the ExternalType("UNKNOWN") stopgap it resolves to today.

• catalog-lakehouse-iceberg: FromIcebergType maps unknown → NullType; ToIcebergType / ToIcebergTypeVisitor map NullType → unknown (via a nullType() dispatch hook, since NullType isn't a PrimitiveType), and reject a required (non-nullable) unknown column per the Iceberg spec.
• docs: unified type reference (Null type section), Iceberg type-mapping table, OpenAPI examples.

No new api type and no common/Python serde change — NullType already exists, round-trips as the JSON token "null", and is already mapped for Spark, Flink, Lance, and CLI (the Python client decodes "null" to NullType with no change).

Builds on the merged native variant support (#11932) and format-version-3 support (#11954).

Fixes #11951

Why are the changes needed?

Loading an Iceberg V3 table with an unknown column through the native API resolves it to ExternalType("UNKNOWN") — opaque (nothing can branch on it), not writable back, and external types have caused downstream problems (unqueryable via Trino #10957; catalogString() written verbatim into DDL #11805). unknown is the universal null/void column type — Iceberg's own converters map it to each engine's null type (Spark NullType, Flink NULL, Arrow null) — and Gravitino already models that as NullType, so this is just the missing wiring.

Does this PR introduce any user-facing change?

Yes — Iceberg V3 tables with an unknown column now load through the native API as null (previously external(UNKNOWN)), and a null-typed column can be written to a format-version-3 Iceberg table. Other connectors are unchanged. Connector propagation (reject-with-test for engines without a null-type equivalent) is a planned follow-up.

How was this patch tested?

• Unit (TestConvertUtil): testUnknownType (converter both directions) and testUnknownColumnToIcebergSchema (write path → optional unknown field; required column rejected).
• Docker IT (CatalogIcebergRestIT, passing) — both cross-surface directions between the Iceberg REST (IRC) API and the native metadata API:
  • testV3TypeConversionViaIcebergClient: IRC writes an unknown column → the native API reads it back as NullType.
  • testCreateUnknownColumnWriteRoundTrip: the native API writes a NullType column at format-version 3 → native reload returns NullType, and the IRC reads the same table back as Iceberg unknown.

[nevzheng]

@mchades @roryqi, could you please take a look when you have a chance? This is ready for review and has been updated with the latest main. Thanks!

[Copilot]

Pull request overview

This PR adds native support in the Iceberg catalog for Iceberg V3’s unknown column type by mapping it to Gravitino’s existing Types.NullType, enabling round-trip conversion through both the native metadata API and the Iceberg REST (IRC) surface.

Changes:

• Add Iceberg unknown ⇄ Gravitino NullType conversions in the Iceberg type converters, including a dedicated visitor dispatch for NullType.
• Enforce Iceberg spec constraints on write: reject required (non-nullable) unknown columns; allow nullable NullType to write as Iceberg unknown (format-version 3).
• Add unit + integration tests and update docs/OpenAPI type references to include null/unknown mapping.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/open-api/datatype.yaml	Adds null to primitive type examples; adjusts OpenAPI examples for type docs.
docs/manage-relational-metadata-using-gravitino.md	Documents NullType semantics and JSON token null.
docs/lakehouse-iceberg-catalog.md	Updates Iceberg type mapping table and documents Null ⇄ Iceberg V3 unknown constraints.
catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java	Adds REST-backend IT coverage for unknown read/write round-trips across native and IRC surfaces.
catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java	Adds unit tests for bidirectional conversion and write-path validation (required unknown rejected).
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergTypeVisitor.java	Adds explicit dispatch hook for Types.NullType (since it is not a PrimitiveType).
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java	Implements NullType → Iceberg UnknownType conversion and enforces optionality for unknown.
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java	Maps Iceberg UNKNOWN to Gravitino Types.NullType.

[github-actions]

Code Coverage Report

Overall Project	67.78% +0.04%	🟢
Files changed	91.18%	🟢

Module	Coverage
aliyun	1.72%	🔴
api	46.91%	🟢
authorization-common	85.96%	🟢
aws	42.04%	🟢
azure	2.47%	🔴
catalog-common	9.92%	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.91%	🟢
catalog-hive	79.42%	🟢
catalog-jdbc-clickhouse	81.46%	🟢
catalog-jdbc-common	44.22%	🟢
catalog-jdbc-doris	81.6%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	80.91%	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	59.18%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.48% +0.67%	🟢
catalog-lakehouse-paimon	84.25%	🟢
catalog-model	77.72%	🟢
cli	44.49%	🟢
client-java	78.31%	🟢
common	51.54%	🟢
core	82.52%	🟢
filesystem-hadoop3	77.3%	🟢
flink	0.0%	🔴
flink-common	47.09%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	12.7%	🔴
hive-metastore-common	53.29%	🟢
iceberg-common	63.63%	🟢
iceberg-rest-server	74.9%	🟢
idp-basic	85.71%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.67%	🔴
lance-rest-server	64.84%	🟢
lineage	53.02%	🟢
optimizer	83.24%	🟢
optimizer-api	21.95%	🔴
server	85.97%	🟢
server-common	74.62%	🟢
spark	28.57%	🔴
spark-common	46.01%	🟢
tencent	69.84%	🟢
trino-connector	40.29%	🟢

Files

Module	File	Coverage
catalog-lakehouse-iceberg	ToIcebergType.java	94.79%	🟢
	FromIcebergType.java	94.12%	🟢
	ToIcebergTypeVisitor.java	69.57%	🟢

[mchades]

please resolve the copilot comment

[mchades]

lgtm. please resolve the conflicts

[nevzheng]

@mchades @roryqi, the conflict with the latest main is resolved in 620ed6c. Local validation passes:

• ./gradlew spotlessApply
• targeted TestConvertUtil
• CatalogIcebergRestIT.testV3TypeConversionViaIcebergClient
• ./gradlew :docs:build

The refreshed GitHub Actions runs are currently marked action_required and need maintainer approval. Could you please approve the workflows and review when available? Thanks!