DROP COLUMN on identity partition field corrupts table metadata and makes it unrecoverable

[haroldjimenez]

Apache Iceberg version

• Iceberg version: 1.10.1
• Spark version: 3.5.8
• Catalog: spark_catalog with Hive Metastore (default)

Query engine

• Spark version: 3.5.8

Please describe the bug 🐞

After performing the correct sequence of DROP PARTITION FIELD followed by DROP COLUMN on an identity partition field, the table enters an unrecoverable state with two cascading failures:

1. Querying .partitions metadata table throws a ValidationException
2. Re-adding the dropped column with the same name throws Cannot create identity partition sourced from different field in schema
3. CALL system.rewrite_manifests() also fails — there is no procedure-based recovery path

The table can only be recovered by dropping and recreating it entirely.
Note: DROP PARTITION FIELD alone works correctly. The issue only occurs when DROP COLUMN follows.

Steps to reproduce

-- 1. Create table with identity partition
CREATE TABLE spark_catalog.default.test_table (
    event_id    BIGINT,
    event_date  DATE,
    event_hour  INT,
    user_id     STRING
)
USING iceberg
PARTITIONED BY (event_date);

-- 2. Insert initial data under spec 0 (event_date only)
INSERT INTO spark_catalog.default.test_table VALUES
    (1, DATE '2024-03-14', 9,  'user_A'),
    (2, DATE '2024-03-15', 10, 'user_B');

-- 3. Add event_hour as identity partition field (spec 1)
ALTER TABLE spark_catalog.default.test_table
ADD PARTITION FIELD event_hour;

-- 4. Insert data under spec 1 (event_date + event_hour)
INSERT INTO spark_catalog.default.test_table VALUES
    (3, DATE '2024-03-16', 14, 'user_C'),
    (4, DATE '2024-03-16', 20, 'user_D');

-- 5. Drop the partition field (works fine)
ALTER TABLE spark_catalog.default.test_table
DROP PARTITION FIELD event_hour;

-- 6. Drop the source column (succeeds with no error)
ALTER TABLE spark_catalog.default.test_table
DROP COLUMN event_hour;

-- 7. Query partitions metadata → CRASH #1
SELECT * FROM spark_catalog.default.test_table.partitions;

-- 8. Try to re-add the column → CRASH #2
ALTER TABLE spark_catalog.default.test_table
ADD COLUMN event_hour INT;

-- 9. Try to recover via rewrite_manifests → CRASH #3
CALL spark_catalog.system.rewrite_manifests(
    table => 'spark_catalog.default.test_table'
);

Expected behavior

• DROP COLUMN should either be blocked with a clear error if old partition specs still reference the column in manifest history, OR the metadata cleanup should handle the column removal gracefully so that .partitions remains queryable and the column can be re-added later.

Related PR

• Handle NPE for VariantLogicalType in TypeWithSchemaVisitor #14261

Actual behavior

Crash #1 — querying .partitions after DROP COLUMN:

org.apache.iceberg.exceptions.ValidationException: Cannot find source column for partition field: 1001: event_hour: identity(1001)
    at org.apache.iceberg.PartitionSpec.checkCompatibility(PartitionSpec.java:661)
    at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:633)
    at org.apache.iceberg.BaseMetadataTable.transformSpec(BaseMetadataTable.java:83)
    at org.apache.iceberg.PartitionsTable.lambda$filteredManifests$4(PartitionsTable.java:230)

Crash #2 — re-adding the column:

java.lang.IllegalArgumentException: Cannot create identity partition sourced from different field in schema: event_hour
    at org.apache.iceberg.PartitionSpec$Builder.checkAndAddPartitionName(PartitionSpec.java:413)
    at org.apache.iceberg.TableMetadata.updateSpecSchema(TableMetadata.java:759)

Crash #3 — rewrite_manifests also fails with the same ValidationException as Crash #1, leaving no procedure-based recovery path.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time