[SPARK-51765][DOCS] Document SQL Scripting Syntax

[davidm-db]

Recreation of #50592, updated to reflect the current state of the feature.

What changes were proposed in this pull request?

Adding documentation for SQL Scripting feature.

Why are the changes needed?

If it is not documented, it never happened :-)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Documentation change, reviewed manually.

Was this patch authored or co-authored using generative AI tooling?

No.

[davidm-db]

cc @cloud-fan @dongjoon-hyun

[dongjoon-hyun]

Thank you for polishing the feature with the documentation and proposing this, @davidm-db .

As a release manager, I'll block this until I re-verify this again. For now, please consider this as -1 because there is not enough time for the community to re-evaluate your suggestion yet.

I don't think this is a blocker for Apache Spark 4.1.0 itself. We can put this and the documentation as a part of Apache Spark 4.2.0 also.

[dongjoon-hyun]

Thank you, @davidm-db .

[dongjoon-hyun]

Merged to master.

There is a conflict on branch-4.1. Could you make a backporting PR to branch-4.1 (if you want to have this in Apache Spark 4.1.0), @davidm-db ?

[davidm-db]

@dongjoon-hyun I created a backport PR to branch-4.1: #53155.

Even if we don't enable Scripting by default in 4.1, there's no reason to not have docs in. At least I think so, because feature can be used by enabling the flag.

I just added a minor note there on how to enable SQL Scripting (i.e. which flag to set), since it's not enabled by default.