Printing execution plan for merge operation with python API

[dexianta]

Hi, is there a way to see the execution plan at this level?
e.g. deltaTable.alias('t1').merge(update_df.alias(t2), 't1.id = t2.id').explain()

I see the example from Databricks was by creating temp view, and executed from SQL, as in spark.sql('EXPLAIN ...')

[zsxwing]

Yep. This sounds a useful API to add.

[stikkireddy]

@zsxwing would something like this work:

  def explain(): Unit = {
    val sparkSession = targetTable.toDF.sparkSession
    val mergeIntoCommand = getMergeIntoCommand(sparkSession)

    // only simple is needed to describe the merge into command
    val explain = ExplainCommand(mergeIntoCommand, mode = ExplainMode.fromString("simple"))
    val plan = SparkSession.active.sessionState.executePlan(explain).executedPlan.executeCollect()
      .map(_.getString(0))
      .mkString("\n")
    // scalastyle:off println
    println(plan)
    // scalastyle:on println
  }

^ based off of spark dataset.explain()...

where the output looks like the following:

== Physical Plan ==
Execute MergeIntoCommand
   +- MergeIntoCommand Project [_1#468 AS key2#473, _2#469 AS value2#474], Relation [key1#477,value1#478] parquet, Delta[version=0, ... :/private/var/folders/gy/dy_2wchs6wz7223r9nzckf980000gq/T/spark-5bd85319-3f6e-491e-9c6b-2cd52ce9481f], (key1#477 = key2#473), [Update [actions: [`key1` = key2#473, `value1` = value2#474]]], [Insert [actions: [`key1` = key2#473, `value1` = value2#474]]], StructType(StructField(key1,IntegerType,true), StructField(value1,IntegerType,true))

Would this be sufficient for explain command?

[zsxwing]

As I mentioned in #910 (comment) , we are not going to build this right now. Closing the ticket.