Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-38322][SQL] Support query stage show runtime statistics in formatted explain mode #35658

Closed
wants to merge 1 commit into from

Conversation

ulysses-you
Copy link
Contributor

What changes were proposed in this pull request?

Add query stage statistics information in formatted explain mode.

Why are the changes needed?

The formatted explalin mode is the powerful explain mode to show the details of query plan. In AQE, the query stage know its statistics if has already materialized. So it can help to quick check the conversion of plan, e.g. join selection.

A simple example:

SELECT * FROM t JOIN t2 ON t.c = t2.c;
== Physical Plan ==
AdaptiveSparkPlan (21)
+- == Final Plan ==
   * SortMergeJoin Inner (13)
   :- * Sort (6)
   :  +- AQEShuffleRead (5)
   :     +- ShuffleQueryStage (4), Statistics(sizeInBytes=16.0 B, rowCount=1)
   :        +- Exchange (3)
   :           +- * Filter (2)
   :              +- Scan hive default.t (1)
   +- * Sort (12)
      +- AQEShuffleRead (11)
         +- ShuffleQueryStage (10), Statistics(sizeInBytes=16.0 B, rowCount=1)
            +- Exchange (9)
               +- * Filter (8)
                  +- Scan hive default.t2 (7)
+- == Initial Plan ==
   SortMergeJoin Inner (20)
   :- Sort (16)
   :  +- Exchange (15)
   :     +- Filter (14)
   :        +- Scan hive default.t (1)
   +- Sort (19)
      +- Exchange (18)
         +- Filter (17)
            +- Scan hive default.t2 (7)

Does this PR introduce any user-facing change?

no, only change the output of explain in AQE

How was this patch tested?

Add test

@github-actions github-actions bot added the SQL label Feb 25, 2022
@ulysses-you
Copy link
Contributor Author

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in 2dc0527 Feb 25, 2022
@ulysses-you
Copy link
Contributor Author

thank you @cloud-fan !

@ulysses-you ulysses-you deleted the exchange-statistics branch February 25, 2022 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants