Skip to content

[SPARK-57938][INFRA] Add new SKIP_ flags to doc build script#57010

Open
nchammas wants to merge 8 commits into
apache:masterfrom
nchammas:build-docs-skip-flags
Open

[SPARK-57938][INFRA] Add new SKIP_ flags to doc build script#57010
nchammas wants to merge 8 commits into
apache:masterfrom
nchammas:build-docs-skip-flags

Conversation

@nchammas

@nchammas nchammas commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a couple of new SKIP_ flags to the existing doc build script:

  • SKIP_SPARK_BUILD
  • SKIP_UNIDOC

Update the docs README to reflect the complete set of skip flags.

Change the docs build script to accept any non-empty value as enabling a particular skip flag.

Why are the changes needed?

#48860 added SPARK_DOCS_IS_BUILT_ON_HOST but it doesn't really fit the existing "interface" of the doc build script, as clunky as it is. (The interface is basically a collection of SKIP_ flags.)

This PR tweaks the new skip flags so they are more consistent with the existing ones.

The change from accepting 1 only to accepting any non-empty value also makes the docs build more consistent with how skip flags are handled elsewhere, like here.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I ran the doc build as follows without issue:

SKIP_UNIDOC=1 SKIP_SPARK_BUILD=1 SKIP_RDOC=1 bundle exec jekyll build

The doc build worked as expected. The Spark build was correctly skipped, and the Scala/Java doc build was quick since it was just copying files around.

I also ran the docs with other values passed in to the skip flags and confirmed they are accepted:

$ SKIP_SPARK_BUILD=true SKIP_RDOC=fart SKIP_SCALADOC=wow SKIP_PYTHONDOC=no SKIP_SQLDOC=yes bundle exec jekyll build
Configuration file: /Users/nicholaschammas/Projects/nchammas/spark/docs/_config.yml

**********************
* Skip flags enabled *
**********************
SKIP_PYTHONDOC
SKIP_RDOC
SKIP_SCALADOC
SKIP_SPARK_BUILD
SKIP_SQLDOC

***********************
* Building error docs *
***********************
Using Python 3.11.9.

<snipped>

Was this patch authored or co-authored using generative AI tooling?

No.

@nchammas

nchammas commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

cc @panbingkun, who authored the referenced PR.

@nchammas nchammas force-pushed the build-docs-skip-flags branch from 69b34c4 to 453ee24 Compare July 5, 2026 19:31
@nchammas nchammas force-pushed the build-docs-skip-flags branch from 453ee24 to 5e74966 Compare July 5, 2026 19:32
Comment thread docs/_plugins/build_api_docs.rb Outdated
cd(SPARK_PROJECT_ROOT)

build_spark_scala_and_java_docs_if_necessary
if not (ENV['SKIP_UNIDOC'] == '1')

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check possible latent inconsistency: SKIP_UNIDOC reuses a name that already exists cross-subsystem; dev/run-tests.py:307 reads it via truthiness and multiple non-doc CI jobs set SKIP_UNIDOC: true, whereas the doc script gates strictly on == '1'. No live conflict (the docs job at lines 1246-1577 sets neither new flag, so it builds both as before, and no CI path both exports SKIP_UNIDOC=true and runs the Jekyll build), but a user exporting SKIP_UNIDOC=true expecting the doc build to skip would be surprised.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the doc build to accept any non-empty value for its skip flags. I've updated the PR description accordingly.

Comment thread docs/README.md Outdated
* `SKIP_SPARK_BUILD=1`: Skip building Spark. A working Spark build is required for the docs (except the error docs) to build correctly, so set this only if you know you've already built Spark.
* `SKIP_API=1`: Skip building all the API docs. This is basically the same as setting all of the skip flags below.
* `SKIP_SCALADOC=1`: Skip the Scala and Java API docs, which includes building the Unidoc.
* `SKIP_UNIDOC=1`: Skip building the Unidoc.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc-wording nit: the SKIP_UNIDOC=1 entry ("Skip building the Unidoc") omits that the Scala/Java copy step (copy_and_update_scala_docs/copy_and_update_java_docs from target/scala-2.13/unidoc and target/javaunidoc) still runs, so the build fails there if no pre-built unidoc output exists. SKIP_SPARK_BUILD=1 carries the analogous "only set this if you've already built" caveat; SKIP_UNIDOC should too (use SKIP_SCALADOC=1 to skip the copy as well).

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @nchammas, this is a clean, well-scoped INFRA refactor that makes the doc-build skip interface consistent and more granular. I left a few nit comments, but otherwise LGTM.

@uros-b uros-b requested a review from panbingkun July 7, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants