[SPARK-46395][DOCS] Assign Spark configs to groups for use in documentation

[nchammas]

What changes were proposed in this pull request?

Enable Spark configs to be assigned to documentation groups via a dedicated YAML file. These groups will be used by a new Python script to automatically build config tables for display in our documentation.

Instead of having to maintain large blocks of HTML tables throughout our documentation, config tables can simply be included as follows:

{% include_api_gen _generated/config_tables/sql-tuning-caching-data.html %}

sql-tuning-caching-data would be the name of the config group defined in the YAML file from which the above HTML table would be automatically generated for inclusion in our documentation.

This approach covers both SQL and non-SQL config docs and, if accepted, will replace sql/gen-sql-config-docs.py.

This proposal is an alternative to #44755 (and #44300) that does not require modifying ConfigEntry or ConfigBuilder to add a new field. Instead, the groups are defined completely outside of Spark's core.

Why are the changes needed?

Using this approach we can accomplish several goals at once:

• Eliminate thousands of lines of manually maintained HTML tables of Spark configs.
• Ensure that internal configs are not accidentally documented publicly. (e.g. spark.sql.files.openCostInBytes)
• Ensure that configs are documented publicly exactly as they are in the code. (e.g. unlike spark.sql.autoBroadcastJoinThreshold)
• Improve the presentation of the HTML tables by refining how long config names wrap, as proposed in [SPARK-46923][DOCS] Limit width of configuration tables #44955.

Does this PR introduce any user-facing change?

No, not for users.

For contributors, the documentation build will change so that config tables are generated only when Spark is built. For fast documentation builds -- like what we currently have via SKIP_API=1 -- Spark won't be built and placeholders will be inserted in the place of every config table. This will be addressed in a future PR.

How was this patch tested?

I manually ran the new script to generate config tables and confirmed the following:

• The desired config tables are generated.
• If a config is mentioned in the YAML file but is not found, the script errors.
• If a config group is defined in the YAML file that uses a reserved name, the script errors.
• If a config group name is invalid, the script errors.

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

@beliefer - Assuming you prefer the approach here to the one in #44755, is there anything else I can do to move this proposal forward? Are you interested in shepherding this work?

There will be many follow-up PRs to incrementally migrate all of our config documentation to use this framework. The result will be that our config documentation is more accurate, more usable, and easier to maintain.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[nchammas]

Closing this in favor of #44755.