[SPARK-30334][SQL] Introduce as_json for marking a column as JSON data

[brkyvz]

What changes were proposed in this pull request?

Semi-structured data is used widely in the data industry for reporting events in a wide variety of formats. Click events in product analytics can be stored as json. Some application logs can be in the form of delimited key=value text. Some data may be in xml.

The goal of this project is to be able to signal Spark that such a column exists. This will then enable Spark to "auto-parse" these columns on the fly. The proposal is to store this information as part of the column metadata, in the fields:

• format: The format of the semi-structured column, e.g. json, xml, avro
• options: Options for parsing these columns

This PR introduces the function "as_json", which accomplishes this for JSON columns.

Why are the changes needed?

Simplify the handling of semi-structured columns in Spark, initially for JSON data.

Does this PR introduce any user-facing change?

Introduces a new function called as_json

How was this patch tested?

Unit tests

[SparkQA]

Test build #115643 has finished for PR 26987 at commit 576af91.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class AsJson(json: Expression, formatOptions: Map[String, String], child: Expression)

[brkyvz]

cc @marmbrus

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!