Native OOM on Large-Scale Joins and Distinct/Deduplication Operations

[arsath7]

1. Context & Environment
   Engine: PySpark with Apache DataFusion Comet plugin enabled.
   Comparison: Vanilla Apache Spark (without Comet native execution).
   Workload Scale: 300 GB+ datasets.

2. Problem Description
   When running scale-benchmarks on 300 GB+ datasets, Apache Comet fails with Out Of Memory (OOM) errors during shuffle-heavy operations (Joins and Distinct/Deduplication).
   In contrast, Vanilla Apache Spark completes the exact same queries successfully, handling memory pressure by spilling intermediate shuffle/aggregate states to disk when executor memory limits are reached (albeit slower).

3. Test Scenarios & Observed Behavior
   Scenario 1: Large-Scale Left Join (~319 GB CSV.GZ + ~30 GB Parquet)
   Operations: Reading a ~319 GB compressed CSV dataset and a ~30 GB Parquet dataset, performing a left outer join on a string ID key, applying several column projections/casts, and writing to Parquet.
   Result:
   Vanilla Spark: Success (spills memory to disk during shuffle/join).
   Comet Engine: OOM Failure during sort/sortMerger.
   Scenario 2: Distinct / Deduplication (~300 GB Parquet)
   Operations: Performing distinct() or dropDuplicates() operations on a single ~300 GB Parquet dataset.
   Result:
   Vanilla Spark: Success.
   Comet Engine: OOM Failure during native aggregation.

4. Troubleshooting & Memory Tuning Attempted
   To rule out configuration misconfigurations, I attempted all standard Comet memory tuning variations as described in the official Apache DataFusion Comet Tuning Guide, including:
   Off-Heap Memory Configurations: Enabled spark.memory.offHeap.enabled=true and progressively scaled spark.memory.offHeap.size.
   Comet Overhead Adjustments: Configured spark.comet.memoryOverhead / spark.comet.memory.overhead.factor to allocate extra non-JVM headroom.
   Memory Pool Configurations: Tested variations of spark.comet.exec.memoryPool (fair_unified vs. greedy_unified).
   Outcome: Despite attempting all tuning options mentioned in the documentation, the workload consistently fails with a native OOM crash when Comet native operators are active.

5. Questions for the Community
   Is native spilling to disk fully supported for DataFusion operators in the latest release?
   Are there specific un-documented or internal configurations required to prevent OOM?