From 17997a57969ed935b543ab8b2af60e944f452f6f Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Wed, 8 Jul 2020 14:11:18 -0700 Subject: [PATCH] [SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins ### What changes were proposed in this pull request? This PR aims to disable SBT `unidoc` generation testing in Jenkins environment because it's flaky in Jenkins environment and not used for the official documentation generation. Also, GitHub Action has the correct test coverage for the official documentation generation. - https://github.com/apache/spark/pull/28848#issuecomment-654577911 (amp-jenkins-worker-06) - https://github.com/apache/spark/pull/28926#issuecomment-654316537 (amp-jenkins-worker-06) - https://github.com/apache/spark/pull/28969#issuecomment-654918636 (amp-jenkins-worker-06) - https://github.com/apache/spark/pull/28975#issuecomment-654447955 (amp-jenkins-worker-05) - https://github.com/apache/spark/pull/28986#issuecomment-654416543 (amp-jenkins-worker-05) - https://github.com/apache/spark/pull/28992#issuecomment-654371469 (amp-jenkins-worker-06) - https://github.com/apache/spark/pull/28993#issuecomment-655289237 (amp-jenkins-worker-05) - https://github.com/apache/spark/pull/28999#issuecomment-653976760 (amp-jenkins-worker-04) - https://github.com/apache/spark/pull/29010#issuecomment-655246083 (amp-jenkins-worker-03) - https://github.com/apache/spark/pull/29013#issuecomment-654292483 (amp-jenkins-worker-04) - https://github.com/apache/spark/pull/29016#issuecomment-654495070 (amp-jenkins-worker-05) - https://github.com/apache/spark/pull/29025#issuecomment-654889938 (amp-jenkins-worker-04) - https://github.com/apache/spark/pull/29042#issuecomment-655587989 (amp-jenkins-worker-03) ### Why are the changes needed? Apache Spark `release-build.sh` generates the official document by using the following command. - https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L341 ```bash PRODUCTION=1 RELEASE_VERSION="$SPARK_VERSION" jekyll build ``` And, this is executed by the following `unidoc` command for Scala/Java API doc. - https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30 ```ruby system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed") ``` However, the PR builder disabled `Jekyll build` and instead has a different test coverage. ```python # determine if docs were changed and if we're inside the amplab environment # note - the below commented out until *all* Jenkins workers can get `jekyll` installed # if "DOCS" in changed_modules and test_env == "amplab_jenkins": # build_spark_documentation() ``` ``` Building Unidoc API Documentation ======================================================================== [info] Building Spark unidoc using SBT with these arguments: -Phadoop-3.2 -Phive-2.3 -Pspark-ganglia-lgpl -Pkubernetes -Pmesos -Phadoop-cloud -Phive -Phive-thriftserver -Pkinesis-asl -Pyarn unidoc ``` ### Does this PR introduce _any_ user-facing change? No. (This is used only for testing and not used in the official doc generation.) ### How was this patch tested? Pass the Jenkins without doc generation invocation. Closes #29017 from dongjoon-hyun/SPARK-DOC-GEN. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- dev/run-tests.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dev/run-tests.py b/dev/run-tests.py index ca502b2818847..223072cbe7bfb 100755 --- a/dev/run-tests.py +++ b/dev/run-tests.py @@ -363,7 +363,8 @@ def build_spark_assembly_sbt(extra_profiles, checkstyle=False): if checkstyle: run_java_style_checks(build_profiles) - build_spark_unidoc_sbt(extra_profiles) + if not os.environ.get("AMPLAB_JENKINS"): + build_spark_unidoc_sbt(extra_profiles) def build_apache_spark(build_tool, extra_profiles):