You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#5370 and #5445 bridge native spill metrics into Spark task metrics (diskBytesSpilled / memoryBytesSpilled), but only for native shuffle write tasks: CometMetricNode.reportSpillMetrics is registered solely on the CometNativeShuffleInputRDD path.
A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via CometExecRDD in operators.scala) still reports spilled_bytes only as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans.
Proposed scope
Register a task-level spill reporting callback for native stages executed via CometExecRDD, aggregating over the stage's CometMetricNode tree as sumMetricValues does for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes).
What / Why
#5370 and #5445 bridge native spill metrics into Spark task metrics (
diskBytesSpilled/memoryBytesSpilled), but only for native shuffle write tasks:CometMetricNode.reportSpillMetricsis registered solely on theCometNativeShuffleInputRDDpath.A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via
CometExecRDDinoperators.scala) still reportsspilled_bytesonly as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans.Proposed scope
CometExecRDD, aggregating over the stage'sCometMetricNodetree assumMetricValuesdoes for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes).