[SPARK-54499][SQL][4.1] Re-enable spark.sql.scripting.enabled by default for Spark 4.1.0

[davidm-db]

What changes were proposed in this pull request?

This PR aims to re-enable SQL Scripting in Spark 4.1, ahead of its RC.

Why are the changes needed?

There was a previous misunderstanding/miscommunication, that I did not notice in time. The feature was disabled because it was thought that it is not ready, due to unclosed JIRA items.

However, it was just a case of not 100% up-to-date items. I have cleaned up and classified remaining work items (children of SPARK-48338) into 4 categories:

• M0 - basic support
• M1 - features and changes required to enable SQL Scripting by default
• M2 - follow-up improvements and additional functionalities that are non-fundamental and should not block M1
• M3 - potential improvements for the future, need investigation

M0 and M1 are done, meaning the feature is stable, useful and also ready to be used.

M2 will improve some of the aspects of using it by introducing a newer and more user-friendly statements, like SIGNAL/RESIGNAL and GET DIAGNOSTICS, as well as do some very minor optimizations. However, without those, feature is still ready to be used as is.

The only missing aspect is documentation, for which we have a PR (#50592) but it was closed. I'll work on re-submitting that one.

Does this PR introduce any user-facing change?

I am not sure 100% how this classifies - the feature was already present in 4.0, but was not enabled by default.
Anyways, the feature is completely orthogonal to all other standalone statement execution paths, so it is just adding to functionality of Spark.
There is no difference in behavior for any standalone SQL statement compared to when SQL Scripting is turned off.

How was this patch tested?

SQL Scripting has a thorough support in tests.
CI ensures the feature does not affect any other.

Was this patch authored or co-authored using generative AI tooling?

No.

[davidm-db]

cc @cloud-fan @dongjoon-hyun @gatorsmile

[cloud-fan]

User can use scripting already by enabling the config, so we need documentation anyway. I don't think this should block 4.1 RC, but if we can finish the documentation before the vote passes, I think re-enabling it is a reasonable choice.

[davidm-db]

but if we can finish the documentation before the vote passes, I think re-enabling it is a reasonable choice

@cloud-fan I just created a PR, based on Serge's previous proposal: #53155

[dongjoon-hyun]

Thank you for polishing the feature with the documentation and proposing this, @davidm-db .

As a release manager, I'll block this until I re-verify this again. For now, please consider this as -1 because there is not enough time for the community to re-evaluate your suggestion yet.

Please note that I agree with @cloud-fan 's comment mostly and I don't think this is a blocker for Apache Spark 4.1.0 itself. We can put this and the documentation as a part of Apache Spark 4.2.0 also.

[dongjoon-hyun]

In addition, please use a different JIRA ID instead of SPARK-54261.

[dongjoon-hyun]

User can use scripting already by enabling the config, so we need documentation anyway.

Just for the record, I reviewed and merged the documentation PR to the master branch for Apache Spark 4.2.0. For branch-4.1, I tried to deliver the documentation but there is a conflict with branch-4.1. So, the documentation is not in branch-4.1 and there is no documentation PR to fit branch-4.1 neither so far.

• [SPARK-51765][DOCS] Document SQL Scripting Syntax #53155 (comment)

[dongjoon-hyun]

In addition, please use a different JIRA ID instead of SPARK-54261.

Please address the review comment to move forward, @davidm-db .

[davidm-db]

@dongjoon-hyun resolved the comment and added a commit to revert the "enablement" section from the docs. sorry, it was middle of the night here when you wrote your last comments here and on the other ticket.

[dongjoon-hyun]

+1, LGTM.

Merged to branch-4.1 for Apache Spark 4.1.0.

Thank you so much for keeping sharing the status and polishing this for 4.1.0, @davidm-db and @cloud-fan .