[SPARK-35198][CONNECT][CORE][PYTHON][SQL] Add support for calling debugCodegen from Python & Java

[juanvisoler]

Add support for calling debugCodegen from Python & Java

What changes were proposed in this pull request?

• Adds debugCodegen to the Dataset APIs core and connector
• Adds debugCodegen to the pyspark dataframe API
• Removes the implicit debugCodegen from sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala as it is public now.

Why are the changes needed?

To add a direct method to get debugCodegen state for Java & Python users of Dataframes.

Does this PR introduce any user-facing change?

debugCodegen is now accessible from all Scala, python and Java APIs. See usage below.

How was this patch tested?

Added tests in:

• sql/core/src/test/scala/org/apache/spark/sql/DebugCodegenSuite.scala.
• connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala

Local testing:

scala> spark.sql("SELECT 1").debugCodegen
Found 1 WholeStageCodegen subtrees.
== Subtree 1 / 1 (maxMethodCodeSize:55; maxConstantPoolSize:99(0.15% used); numInnerClasses:0) ==
*(1) Project [1 AS 1#0]
+- *(1) Scan OneRowRelation[]
...

>>> spark.sql("select 1").debugCodegen()
Found 1 WholeStageCodegen subtrees.
== Subtree 1 / 1 (maxMethodCodeSize:55; maxConstantPoolSize:99(0.15% used); numInnerClasses:0) ==
*(1) Project [1 AS 1#0]
+- *(1) Scan OneRowRelation[]
...

[juanvisoler]

@holdenk @MaxGekk

[arturobernalg]

LGTM +1

[juanvisoler]

@HyukjinKwon Changed the implementation to df.explain("codegen") and added it to the connector

[juanvisoler]

Hi @HyukjinKwon Does the PR approach make sense to you?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[juanvisoler]

Hey @HyukjinKwon @MaxGekk @holdenk, I created this PR a while ago... I would appreciate if you have the chance to review it.
Thanks!

[holdenk]

LGTM for 4.0, can you update the version tags and then we'll merge to the dev branch?

[juanvisoler]

LGTM for 4.0, can you update the version tags and then we'll merge to the dev branch?

Thanks @holdenk. Updated the version tags to 4.0.0 but the bot closed the PR first. It should show up when you reopen the PR.

[arturobernalg]

LGTM for 4.0, can you update the version tags and then we'll merge to the dev branch?

Thanks @holdenk. Updated the version tags to 4.0.0 but the bot closed the PR first. It should show up when you reopen the PR.

LGTM

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!