perf: [iceberg] per partition file scan task serialization

[parthchandra]

Experimental to see if this approach to do per partition file scan task serialization works.

Note: Could not have been done without Claude Code.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 43.61702% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.84%. Comparing base (f09f8af) to head (859d3a8).
⚠️ Report is 901 commits behind head on main.

Files with missing lines	Patch %	Lines
...e/spark/sql/comet/CometIcebergNativeScanExec.scala	40.90%	25 Missing and 1 partial ⚠️
...la/org/apache/spark/sql/comet/IcebergScanRDD.scala	0.00%	19 Missing ⚠️
spark/src/main/scala/org/apache/comet/Native.scala	0.00%	5 Missing ⚠️
...rc/main/java/org/apache/comet/NativeJNIBridge.java	0.00%	2 Missing ⚠️
.../comet/serde/operator/CometIcebergNativeScan.scala	95.45%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3301      +/-   ##
============================================
+ Coverage     56.12%   59.84%   +3.71%     
- Complexity      976     1482     +506     
============================================
  Files           119      177      +58     
  Lines         11743    16256    +4513     
  Branches       2251     2685     +434     
============================================
+ Hits           6591     9728    +3137     
- Misses         4012     5179    +1167     
- Partials       1140     1349     +209     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[parthchandra]

Closing as this did not address the problem