[SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing#55787
Open
yaooqinn wants to merge 1 commit into
Open
[SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing#55787yaooqinn wants to merge 1 commit into
yaooqinn wants to merge 1 commit into
Conversation
### What changes were proposed in this pull request? This PR restores sub-execution grouping on the SQL tab listing that was silently dropped when the listing was switched to server-side pagination in SPARK-56140. - Backend (`SqlResource.sqlTable`): accept a new `groupSubExecution` query parameter (default = cluster config `spark.ui.groupSQLSubExecutionEnabled`). When enabled, paginate over root executions only and embed each root's children as `subExecutions: [...]` on the row. Sub-executions whose root is missing from the filtered set surface as roots so they are never hidden. Fix `recordsTotal` / `recordsFiltered` to count root rows in grouped mode so the DataTables "Showing X to Y of Z entries" matches the visible rows. - Frontend (`allexecutionspage.js`): read the existing `group-sub-exec-config` data attribute, forward `groupSubExecution=true|false` to the server, and append a trailing **Sub Executions** column showing a `+N sub` toggle on roots that have children. Toggling expands the row using DataTables `row().child()` with a nested table — matches the SPARK-41752 / Spark 4.1 layout. - CSS (`webui-dataTables.css`): minimal styling for the toggle link and the nested child table (indent + tertiary background). ### Why are the changes needed? SPARK-41752 introduced sub-execution grouping in Spark 4.1, and SPARK-55875 carried it over when the listing moved to client-side DataTables. SPARK-56140 then switched the listing to server-side pagination but didn't carry over the grouping logic — every sub-execution now shows up as its own flat row, regressing the UX for queries such as `CACHE TABLE` and nested CTAS. ### Does this PR introduce _any_ user-facing change? Yes — the SQL tab listing again folds sub-executions under their root, with a `+N sub` toggle to expand. Default behaviour is controlled by the existing `spark.ui.groupSQLSubExecutionEnabled` config (default true). ### How was this patch tested? - New unit test in `SqlResourceWithActualMetricsSuite` covering both grouped and flat modes against a session that runs CACHE TABLE (which produces a root + sub-execution pair). - Existing `SqlResourceWithActualMetricsSuite` and `AllExecutionsPageWithInMemoryStoreSuite` continue to pass. - Manual verification in a local Spark UI: ran a workload with two sub-execution-producing statements (CACHE TABLE, CTAS) and confirmed both modes render correctly via screenshot. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR restores sub-execution grouping on the SQL tab listing that was silently dropped when the listing was switched to server-side pagination in SPARK-56140.
SqlResource.sqlTable): accept a newgroupSubExecutionquery parameter (default = cluster configspark.ui.groupSQLSubExecutionEnabled). When enabled, paginate over root executions only and embed each root's children assubExecutions: [...]on the row. Sub-executions whose root is missing from the filtered set surface as roots so they are never hidden. FixrecordsTotal/recordsFilteredto count root rows in grouped mode so the DataTables "Showing X to Y of Z entries" matches the visible rows.allexecutionspage.js): read the existinggroup-sub-exec-configdata attribute, forwardgroupSubExecution=true|falseto the server, and append a trailing Sub Executions column showing a+N subtoggle on roots that have children. Toggling expands the row using DataTablesrow().child()with a nested table — matches the SPARK-41752 / Spark 4.1 layout.webui-dataTables.css): minimal styling for the toggle link and the nested child table (indent + tertiary background).Why are the changes needed?
SPARK-41752 introduced sub-execution grouping in Spark 4.1, and SPARK-55875 carried it over when the listing moved to client-side DataTables. SPARK-56140 then switched the listing to server-side pagination but didn't carry over the grouping logic — every sub-execution now shows up as its own flat row, regressing the UX for queries such as
CACHE TABLEand nested CTAS.Does this PR introduce any user-facing change?
Yes — the SQL tab listing again folds sub-executions under their root, with a
+N subtoggle to expand. Default behaviour is controlled by the existingspark.ui.groupSQLSubExecutionEnabledconfig (defaulttrue).The screenshots below were taken from the same workload (
SELECT,CACHE TABLE, broadcast join,DROP TABLE,CREATE TABLE … AS) —CACHE TABLEand the innerCREATE TABLEeach spawn a sub-execution.Before — flat listing (today, post-SPARK-56140): sub-executions appear as flat sibling rows; e.g. id 2/3/4 are all
CACHE TABLE people_eng_cachedand id 7/8/9 are allnested CTAS.After — grouped, collapsed: only roots are listed, with a trailing
+1 subtoggle on rows that own a sub-execution.After — grouped, expanded: clicking
+1 subexpands a nested table inline.How was this patch tested?
SqlResourceWithActualMetricsSuitecovering both grouped and flat modes against a session that runsCACHE TABLE(which produces a root + sub-execution pair).SqlResourceWithActualMetricsSuiteandAllExecutionsPageWithInMemoryStoreSuitecontinue to pass.spark.ui.groupSQLSubExecutionEnabled=false(flat) and once with the defaulttrue(grouped) — and confirmed both modes render correctly.Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning)