Skip to content

[SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing#55787

Open
yaooqinn wants to merge 1 commit into
apache:masterfrom
yaooqinn:SPARK-56811
Open

[SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing#55787
yaooqinn wants to merge 1 commit into
apache:masterfrom
yaooqinn:SPARK-56811

Conversation

@yaooqinn
Copy link
Copy Markdown
Member

@yaooqinn yaooqinn commented May 10, 2026

What changes were proposed in this pull request?

This PR restores sub-execution grouping on the SQL tab listing that was silently dropped when the listing was switched to server-side pagination in SPARK-56140.

  • Backend (SqlResource.sqlTable): accept a new groupSubExecution query parameter (default = cluster config spark.ui.groupSQLSubExecutionEnabled). When enabled, paginate over root executions only and embed each root's children as subExecutions: [...] on the row. Sub-executions whose root is missing from the filtered set surface as roots so they are never hidden. Fix recordsTotal / recordsFiltered to count root rows in grouped mode so the DataTables "Showing X to Y of Z entries" matches the visible rows.
  • Frontend (allexecutionspage.js): read the existing group-sub-exec-config data attribute, forward groupSubExecution=true|false to the server, and append a trailing Sub Executions column showing a +N sub toggle on roots that have children. Toggling expands the row using DataTables row().child() with a nested table — matches the SPARK-41752 / Spark 4.1 layout.
  • CSS (webui-dataTables.css): minimal styling for the toggle link and the nested child table (indent + tertiary background).

Why are the changes needed?

SPARK-41752 introduced sub-execution grouping in Spark 4.1, and SPARK-55875 carried it over when the listing moved to client-side DataTables. SPARK-56140 then switched the listing to server-side pagination but didn't carry over the grouping logic — every sub-execution now shows up as its own flat row, regressing the UX for queries such as CACHE TABLE and nested CTAS.

Does this PR introduce any user-facing change?

Yes — the SQL tab listing again folds sub-executions under their root, with a +N sub toggle to expand. Default behaviour is controlled by the existing spark.ui.groupSQLSubExecutionEnabled config (default true).

The screenshots below were taken from the same workload (SELECT, CACHE TABLE, broadcast join, DROP TABLE, CREATE TABLE … AS) — CACHE TABLE and the inner CREATE TABLE each spawn a sub-execution.

Before — flat listing (today, post-SPARK-56140): sub-executions appear as flat sibling rows; e.g. id 2/3/4 are all CACHE TABLE people_eng_cached and id 7/8/9 are all nested CTAS.

before

After — grouped, collapsed: only roots are listed, with a trailing +1 sub toggle on rows that own a sub-execution.

after-collapsed

After — grouped, expanded: clicking +1 sub expands a nested table inline.

after-expanded

How was this patch tested?

  • New unit test in SqlResourceWithActualMetricsSuite covering both grouped and flat modes against a session that runs CACHE TABLE (which produces a root + sub-execution pair).
  • Existing SqlResourceWithActualMetricsSuite and AllExecutionsPageWithInMemoryStoreSuite continue to pass.
  • Manual verification in a local Spark UI (screenshots above): ran the same workload twice — once with spark.ui.groupSQLSubExecutionEnabled=false (flat) and once with the default true (grouped) — and confirmed both modes render correctly.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning)

### What changes were proposed in this pull request?

This PR restores sub-execution grouping on the SQL tab listing that was
silently dropped when the listing was switched to server-side pagination
in SPARK-56140.

- Backend (`SqlResource.sqlTable`): accept a new `groupSubExecution`
  query parameter (default = cluster config
  `spark.ui.groupSQLSubExecutionEnabled`). When enabled, paginate over
  root executions only and embed each root's children as
  `subExecutions: [...]` on the row. Sub-executions whose root is
  missing from the filtered set surface as roots so they are never
  hidden. Fix `recordsTotal` / `recordsFiltered` to count root rows
  in grouped mode so the DataTables "Showing X to Y of Z entries"
  matches the visible rows.
- Frontend (`allexecutionspage.js`): read the existing
  `group-sub-exec-config` data attribute, forward
  `groupSubExecution=true|false` to the server, and append a trailing
  **Sub Executions** column showing a `+N sub` toggle on roots that
  have children. Toggling expands the row using DataTables
  `row().child()` with a nested table — matches the SPARK-41752 /
  Spark 4.1 layout.
- CSS (`webui-dataTables.css`): minimal styling for the toggle link
  and the nested child table (indent + tertiary background).

### Why are the changes needed?

SPARK-41752 introduced sub-execution grouping in Spark 4.1, and
SPARK-55875 carried it over when the listing moved to client-side
DataTables. SPARK-56140 then switched the listing to server-side
pagination but didn't carry over the grouping logic — every
sub-execution now shows up as its own flat row, regressing the UX
for queries such as `CACHE TABLE` and nested CTAS.

### Does this PR introduce _any_ user-facing change?

Yes — the SQL tab listing again folds sub-executions under their
root, with a `+N sub` toggle to expand. Default behaviour is
controlled by the existing `spark.ui.groupSQLSubExecutionEnabled`
config (default true).

### How was this patch tested?

- New unit test in `SqlResourceWithActualMetricsSuite` covering both
  grouped and flat modes against a session that runs CACHE TABLE
  (which produces a root + sub-execution pair).
- Existing `SqlResourceWithActualMetricsSuite` and
  `AllExecutionsPageWithInMemoryStoreSuite` continue to pass.
- Manual verification in a local Spark UI: ran a workload with two
  sub-execution-producing statements (CACHE TABLE, CTAS) and confirmed
  both modes render correctly via screenshot.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant