Native Iceberg write panics in construct_partition_summaries after partition spec evolution

[andygrove]

Describe the bug

Writing to a table whose partition spec has been evolved (a partition field and its source column dropped) panics inside iceberg-rust while Comet encodes the manifest:

Comet native panic: panicked at itertools-0.13.0/src/zip_eq_impl.rs:49:17:
itertools: .zip_eq() reached end of one iterator before the other
    at std::panicking::begin_panic::<&str>
    at <iceberg::spec::manifest::writer::ManifestWriter>::construct_partition_summaries
    at comet::execution::operators::iceberg_write::encode_data_files_as_manifest::{closure#0}
    at <comet::execution::operators::iceberg_write::IcebergWriteExec as ...ExecutionPlan>::execute::{closure#0}

surfacing on the JVM side as org.apache.comet.CometNativeException: itertools: .zip_eq() reached end of one iterator before the other, raised through CometIcebergWriteExec.drainAvroPayload.

construct_partition_summaries zips the partition values of each data file against the fields of a partition spec. After the spec has been evolved, those two lengths disagree and zip_eq panics rather than returning an error.

Two problems here: the write fails at all, and it fails by panicking across the JNI boundary rather than by returning a Result.

Steps to reproduce

Spark 4.1.3, Iceberg 1.11.0, spark.comet.iceberg.write.enabled=true plus the Iceberg Spark SQL test setup from dev/diffs/iceberg/1.11.0.diff:

./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 -DflinkVersions= -DkafkaVersions= \
  :iceberg-spark:iceberg-spark-extensions-4.1_2.13:test \
  --tests '*TestAlterTablePartitionFields*' -Pquick=true -x javadoc

Failing cases, all of which drop a partition field and then its source column before writing again:

• testDropPartitionAndSourceColumnLong
• testDropPartitionAndSourceColumnTimestamp
• deleteAfterDroppingPartitionAndSourceColumn

Expected behavior

Writing to a table with an evolved partition spec succeeds and produces the same manifests as iceberg-java. Failing that, the native writer returns an error that Comet can turn into a fallback or a clean Spark exception, and never panics.

Additional context

Found by turning the two Iceberg write flags on by default in #5677.

Shares a trigger with the sibling failure in the same test class filed separately: testReaddColumnAfterIdentityPartitionDrop fails with No column with source column id 2 in schema from the same encode_data_files_as_manifest phase. A single fix to how the writer resolves the partition spec for already-written data files may close both.

Part of #5649.