[SPARK-47289][SQL] Allow extensions to log extended information in explain plan #45488

parthchandra · 2024-03-13T00:32:09Z

What changes were proposed in this pull request?

This addresses SPARK-47289 and adds a new section in explain plan where Spark extensions can add additional information for end users. The section is included in the output only if the relevant configuration is enabled and if the extension actually adds some new information

Why are the changes needed?

Extensions to Spark can add their own planning rules and sometimes may need to add additional information about how the plan was generated. This is useful for end users in determining if the extensions rules are working as intended.

Does this PR introduce any user-facing change?

This PR increases the information logged in the UI in the query plan. The attached screenshot shows output from an extension which provides some of its own operations but does not support some operations.

How was this patch tested?

Unit test and manual testing

Was this patch authored or co-authored using generative AI tooling?

No

dongjoon-hyun

Hi, @parthchandra .

Please enable GitHub Action in your repository.

https://github.com/apache/spark/pull/45488/checks?check_run_id=22588243157

parthchandra · 2024-03-13T17:15:58Z

@dongjoon-hyun github actions are enabled in my repository and the branch is based on the latest commit in master. In my repo the ci checks are shown as passing.

I've never encountered this situation in github before. Any suggestions?

…plain plan

parthchandra · 2024-03-14T19:40:56Z

Rebased on latest master and tests are passing now

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

dongjoon-hyun

Thank you for updating. +1, LGTM (Pending CIs).

cc @cloud-fan , @HyukjinKwon , too

cloud-fan · 2024-03-22T04:22:05Z

sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala

@@ -583,6 +611,24 @@ case class MyParser(spark: SparkSession, delegate: ParserInterface) extends Pars
    delegate.parseQuery(sqlText)
 }

+case class ExplainPlanInfoTagRule() extends Rule[SparkPlan] {


This is really a hacky way to provide extended EXPLAIN info. Can we add a interface like

trait ExtendedExplainGenerator { def generateExtendedInfo(plan: SparkPlan): String }

Then the config provides the implementation class.

Let me see if I understand this right -
A session extension implementation can implement the trait above and set the implementation in a config. If such an implementation is provided, then explain plan can print it. We can replace the enableExtensionInfo flag with this new conf.
The contents and formatting of the extended explain string etc, now become part of the session extension and so we can remove the changes in QueryPlan. The extension implementors will have to do more work, but likely will have greater flexibility.
Let me know if I misunderstood. Otherwise I'll go ahead with this change.

Thank you for the review, @cloud-fan .

Yes, the EXPLAIN command will call this plugin implementation and append the result to the command output as extended information. Then we don't need to add new APIs in QueryPlan and the plugin can also be more flexible as it sees the full plan tree.

parthchandra · 2024-03-22T16:28:53Z

CI test failure appears to be unrelated. Not re-running, because this PR will change.

parthchandra · 2024-04-02T18:21:57Z

@cloud-fan I've update this PR. Can you please take a look?
I also moved the unit test to a more appropriate test suite.
(Sorry it took a while; I was making sure that with this approach extensions would be able to publish their extended info and also be able to build and work with older versions of Spark which do not have the trait.)

dongjoon-hyun · 2024-04-02T18:38:43Z

sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala

 import scala.io.Source
-


nit. Could you run dev/lint-scala?

dongjoon-hyun · 2024-04-02T18:39:21Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExtendedExplainGenerator.scala

+ * A trait for a session extension to implement that provides addition explain plan
+ * information.
+ */
+


nit. Remove empty line.

dongjoon-hyun · 2024-04-02T18:41:44Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

+        try {
+          val extensionClass = Utils.classForName(generator)
+          val extension = extensionClass.getConstructor().newInstance()
+            .asInstanceOf[ExtendedExplainGenerator]


Just a question. Do you know how many times we need to generate?

I don't think it gets called too often. It should only get called in explain plan or when the plan is displayed in the UI.
I can make this a lazy val and load it just once in the class if you think that is preferable.

dongjoon-hyun · 2024-04-02T19:26:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -333,6 +333,13 @@ object SQLConf {
    .booleanConf
    .createWithDefault(true)

+  val EXTENDED_EXPLAIN_PROVIDER = buildConf("spark.sql.extendedExplainProvider")


StaticSQLConf might be a better place for this conf. We don't want to change this per query, right?

Moved to static conf

cloud-fan · 2024-04-03T08:35:15Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExtendedExplainGenerator.scala

+ * A trait for a session extension to implement that provides addition explain plan
+ * information.
+ */
+trait ExtendedExplainGenerator {


let's add

@DeveloperApi @Since("4.0.0")

Good point. Thanks!

cloud-fan · 2024-04-03T08:35:48Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExtendedExplainGenerator.scala

+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution


This is an internal package. Let's put it in org.apache.spark.sql

cloud-fan · 2024-04-03T08:36:38Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

@@ -369,6 +372,26 @@ class QueryExecution(
    Utils.redact(sparkSession.sessionState.conf.stringRedactionPattern, message)
  }

+  def extendedExplainInfo(append: String => Unit, plan: SparkPlan): Unit = {
+    try {
+      val generator = sparkSession.sparkContext.conf.get(EXTENDED_EXPLAIN_PROVIDER.key)


looking at the implementation, seems the config can be a session config instead of static?

It was originally a session config, then after a previous review comment, made it static. Thinking about it again, it seems it might be better to allow it as a session conf. The extended info can then be turned on/off for specific sessions among other cases.

cloud-fan · 2024-04-03T08:38:08Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

+      try {
+        val extensionClass = Utils.classForName(generator)
+        val extension = extensionClass.getConstructor().newInstance()
+          .asInstanceOf[ExtendedExplainGenerator]


Let's call Utils.loadExtensions

Good suggestion. I was not aware of this method. It is much cleaner to use.
Ironically, we do not use this method to load session extensions!

cloud-fan · 2024-04-03T08:41:30Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

+      } catch {
+        case e@(_: ClassCastException |
+                _: ClassNotFoundException |
+                _: NoClassDefFoundError) =>


how about catch NonFatal?

… SQLConf

cloud-fan · 2024-04-04T05:46:18Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

+      generators,
+      sparkSession.sparkContext.conf)
+    if (extensions.nonEmpty) {
+      append("\n== Extended Information ==\n")


Since we support more than one ExtendedExplainGenerator, shall we add a new method def title: String in the trait and included it here like == Extended Information ($title) ==?

Great idea - done.

dongjoon-hyun · 2024-04-04T14:47:31Z

BTW, connect-client-jvm seems to start to fail in CIs.

Do connect-client-jvm module mima check ...
finish connect-client-jvm module mima check ...
ERROR: Comparing Client jar: /__w/spark/spark/connector/connect/client/jvm/target/scala-2.13/spark-connect-client-jvm_2.13-4.0.0-SNAPSHOT.jar and Sql jar: /__w/spark/spark/sql/core/target/scala-2.13/spark-sql_2.13-4.0.0-SNAPSHOT.jar 
problems with Sql module: 
interface org.apache.spark.sql.ExtendedExplainGenerator does not have a correspondent in client version
Exceptions to binary compatibility can be added in 'CheckConnectJvmClientCompatibility#checkMiMaCompatibilityWithSqlModule'
connect-client-jvm module mima check failed.

parthchandra · 2024-04-04T19:51:37Z

@dongjoon-hyun thank you for pointing the ci failure out. I've addressed it by excluding the newly added trait from the compatibility check (it is not needed by the client).

cloud-fan · 2024-04-05T02:34:36Z

thanks, merging to master!

parthchandra · 2024-04-05T03:16:28Z

Thank you @cloud-fan @dongjoon-hyun for the review!

dongjoon-hyun · 2024-04-05T15:03:01Z

Thank you, @parthchandra and @cloud-fan .
This is much better. :)

github-actions bot added the SQL label Mar 13, 2024

dongjoon-hyun reviewed Mar 13, 2024

View reviewed changes

[SPARK-47289][SQL] Allow extensions to log extended information in ex…

94d1b40

…plain plan

parthchandra force-pushed the explain_extensions branch from db309c7 to 94d1b40 Compare March 13, 2024 22:25