chore: Add microbenchmark for IcebergScan operator serde roundtrip

[andygrove]

Summary

This PR adds a microbenchmark for measuring the serialization/deserialization performance of Iceberg FileScanTask objects to protobuf.

The benchmark:

• Creates a real Iceberg table with configurable number of partitions (default: 30,000)
• Extracts actual FileScanTask objects through query planning
• Benchmarks conversion from FileScanTask to Protobuf via CometIcebergNativeScan.convert()
• Benchmarks serialization to bytes and deserialization

Usage

# Run with default 30000 partitions
make benchmark-org.apache.spark.sql.benchmark.CometOperatorSerdeBenchmark

# Run with custom partition count
make benchmark-org.apache.spark.sql.benchmark.CometOperatorSerdeBenchmark -- 1000

Sample Results (1000 partitions)

IcebergScan serde (1000 partitions, 1000 tasks):    Best Time(ms)   Avg Time(ms)   Relative
-------------------------------------------------------------------------------------------
FileScanTask -> Protobuf (convert)                           1043           1058       1.0X
FileScanTask -> Protobuf -> bytes                            1126           1133       0.9X
bytes -> Protobuf (parseFrom)                                  10             11     107.7X
Full roundtrip (convert + serialize + deserialize)           1150           1159       0.9X

Key insight: The conversion from FileScanTask to protobuf dominates (~99% of time). Protobuf parsing is extremely fast.

Serialized size: 178.7 KB for 1000 tasks (~179 bytes/task)

Test plan

• Benchmark compiles and runs successfully
• Results are consistent across multiple runs

🤖 Generated with Claude Code

[mbutrovich]

Is this true? Often when I write in batches to Iceberg tables I get different files for iterations of inserts, and then need to do a compaction after.

[mbutrovich]

The table has numPartitions partitions, right. But does the query run with that many? I figured Spark would use however many shuffle partitions it's configured for, and if num shuffle partitions < num table partitions, table partitions get grouped together. Could you confirm what you mean here? For serde it might be interesting to measure Spark partitions, not table partitions. I just want to make sure we're measuring what we expect here.

[mbutrovich]

Thanks @andygrove. Let's merge this so we can quantify the serialization efforts we're doing. This is a good foundation.