Skip to content

Report native operator spill metrics in Spark task metrics for non-shuffle stages #5447

Description

@peterxcli

What / Why

#5370 and #5445 bridge native spill metrics into Spark task metrics (diskBytesSpilled / memoryBytesSpilled), but only for native shuffle write tasks: CometMetricNode.reportSpillMetrics is registered solely on the CometNativeShuffleInputRDD path.

A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via CometExecRDD in operators.scala) still reports spilled_bytes only as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans.

Proposed scope

  • Register a task-level spill reporting callback for native stages executed via CometExecRDD, aggregating over the stage's CometMetricNode tree as sumMetricValues does for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes).
  • Register it before the native iterator so final metrics are published first, including for failed attempts, matching the ordering established in fix: report native child spill metrics in shuffle tasks #5445.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions