[SPARK-32245][INFRA] Run Spark tests in Github Actions #29057

HyukjinKwon · 2020-07-09T14:08:31Z

What changes were proposed in this pull request?

This PR aims to run the Spark tests in Github Actions.

To briefly explain the main idea:

Reuse dev/run-tests.py with SBT build
Reuse the modules in dev/sparktestsupport/modules.py to test each module
Pass the modules to test into dev/run-tests.py directly via TEST_ONLY_MODULES environment variable. For example, pyspark-sql,core,sql,hive.
dev/run-tests.py does not take the dependent modules into account but solely the specified modules to test.

Another thing to note might be SlowHiveTest annotation. Running the tests in Hive modules takes too much so the slow tests are extracted and it runs as a separate job. It was extracted from the actual elapsed time in Jenkins:

So, Hive tests are separated into to jobs. One is slow test cases, and the other one is the other test cases.

Note that the current GitHub Actions build virtually copies what the default PR builder on Jenkins does (without other profiles such as JDK 11, Hadoop 2, etc.). The only exception is Kinesis https://github.com/apache/spark/pull/29057/files#diff-04eb107ee163a50b61281ca08f4e4c7bR23

Why are the changes needed?

Last week and onwards, the Jenkins machines became very unstable for many reasons:

Apparently, the machines became extremely slow. Almost all tests can't pass.
One machine (worker 4) started to have the corrupt .m2 which fails the build.
Documentation build fails time to time for an unknown reason in Jenkins machine specifically. This is disabled for now at [SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017.
Almost all PRs are basically blocked by this instability currently.

The advantages of using Github Actions:

To avoid depending on few persons who can access to the cluster.
To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce.
To control the environment more flexibly.
Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost.

Note that:

The current build in Jenkins takes more than 7 hours. With Github actions it takes less than 2 hours
We can now control the environments especially for Python easily.
The test and build look more stable than the Jenkins'.

Does this PR introduce any user-facing change?

No, dev-only change.

How was this patch tested?

Tested at HyukjinKwon#4

HyukjinKwon · 2020-07-09T14:14:38Z

dev/run-pip-tests

      conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
-      source activate "$VIRTUALENV_PATH"
+      conda activate "$VIRTUALENV_PATH"


conda activate seems the recommended way, https://docs.conda.io/projects/conda/en/latest/release-notes.html#id203

With conda 4.4, conda activate and conda deactivate are now the preferred commands for activating and deactivating environments.

HyukjinKwon · 2020-07-09T14:14:59Z

dev/run-pip-tests

@@ -76,8 +76,12 @@ for python in "${PYTHON_EXECS[@]}"; do
    VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
    rm -rf "$VIRTUALENV_PATH"
    if [ -n "$USE_CONDA" ]; then
+      if [ -f "$CONDA_PREFIX/etc/profile.d/conda.sh" ]; then
+        # See also https://github.com/conda/conda/issues/7980
+        source $CONDA_PREFIX/etc/profile.d/conda.sh


This is required for Github Actions build. See the link in the comment.

HyukjinKwon · 2020-07-09T14:16:10Z

project/SparkBuild.scala

@@ -1015,9 +1015,20 @@ object TestSettings {
      sys.props.get("test.exclude.tags").map { tags =>
        Seq("--exclude-categories=" + tags)
      }.getOrElse(Nil): _*),
+    // Include tags defined in a system property


This is rather a followup of SPARK-24711.

HyukjinKwon · 2020-07-09T14:16:45Z

project/SparkBuild.scala

    // Show full stack trace and duration in test cases.
    testOptions in Test += Tests.Argument("-oDF"),
    testOptions in Test += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+    // Required to detect Junit tests for each project, see also https://github.com/sbt/junit-interface/issues/35
+    crossPaths := false,


This is required to run the Junit tests when you run the tests of the specified module. For example, ./build/sbt core/test.

HyukjinKwon · 2020-07-09T14:17:19Z

python/pyspark/sql/tests/test_arrow.py

-        self.assertIsInstance(arrow_first_category_element, str)
-        self.assertIsInstance(spark_first_category_element, str)
+        self.assertIsInstance(arrow_first_category_element, basestring)
+        self.assertIsInstance(spark_first_category_element, basestring)


This is rather a followup of SPARK-25351. We didn't drop Python 2 yet.

HyukjinKwon · 2020-07-09T14:18:49Z

python/pyspark/sql/tests/test_types.py

@@ -725,7 +725,8 @@ def assertCollectSuccess(typecode, value):
        if sys.version_info[0] < 3:
            all_types = set(['c', 'b', 'B', 'u', 'h', 'H', 'i', 'I', 'l', 'L', 'f', 'd'])
        else:
-            all_types = set(array.typecodes)
+            # PyPy seems not having array.typecodes.
+            all_types = set(['b', 'B', 'u', 'h', 'H', 'i', 'I', 'l', 'L', 'q', 'Q', 'f', 'd'])


This is also rather a followup of SPARK-22566

HyukjinKwon · 2020-07-09T14:48:53Z

.github/workflows/master.yml

@@ -9,148 +9,233 @@ on:
    - master


Oh, BTW, @dongjoon-hyun. I happened to remove the JDK 11 builds here. I filed a JIRA to recover it at SPARK-32248. I promise I will take a look right away after this PR is merged.

gengliangwang · 2020-07-09T14:49:24Z

on:
  push:
    branches:	   
    - master

I think we should disable the on push trigger if we are running all jobs

dongjoon-hyun

It's great, @HyukjinKwon . I'm okay with the removal of JDK11 build . Yes, we can add that later via SPARK-32248.

.github/workflows/master.yml

python/run-tests.py

HyukjinKwon · 2020-07-09T16:31:23Z

Just for the record, all tests passed at here https://github.com/HyukjinKwon/spark/runs/854407596, and it took 1h 47m 25s.

dev/run-tests.py

common/tags/src/test/java/org/apache/spark/tags/HeavyHiveTest.java

viirya · 2020-07-09T23:52:03Z

.github/workflows/master.yml

+          - hadoop3.2
+        hive:
+          - hive2.3


hive-1.2 and hadoop-2.7 are also removed only for now?

Yeah, I filed a JIRA to restore it at SPARK-32255. I was thinking we can periodically run. The current PR mimics our default PR builder behaviour for now.

We can change hive and hadoop profile on Jenkins test by changing the PR title. I think it is not possible with Github Actions?

Yeah, I think this isn't possible at this moment. It would be great if we can figure out a way to do it.

.github/workflows/master.yml

HyukjinKwon · 2020-07-10T01:55:50Z

I believe now it's good to go when all the tests pass.

This PR aims to run the Spark tests in Github Actions. To briefly explain the main idea: - Reuse `dev/run-tests.py` with SBT build - Reuse the modules in `dev/sparktestsupport/modules.py` to test each module - Pass the modules to test into `dev/run-tests.py` directly via `TEST_ONLY_MODULES` environment variable. For example, `pyspark-sql,core,sql,hive`. - `dev/run-tests.py` _does not_ take the dependent modules into account but solely the specified modules to test. Another thing to note might be `SlowHiveTest` annotation. Running the tests in Hive modules takes too much so the slow tests are extracted and it runs as a separate job. It was extracted from the actual elapsed time in Jenkins: ![Screen Shot 2020-07-09 at 7 48 13 PM](https://user-images.githubusercontent.com/6477701/87050238-f6098e80-c238-11ea-9c4a-ab505af61381.png) So, Hive tests are separated into to jobs. One is slow test cases, and the other one is the other test cases. _Note that_ the current GitHub Actions build virtually copies what the default PR builder on Jenkins does (without other profiles such as JDK 11, Hadoop 2, etc.). The only exception is Kinesis https://github.com/apache/spark/pull/29057/files#diff-04eb107ee163a50b61281ca08f4e4c7bR23 Last week and onwards, the Jenkins machines became very unstable for many reasons: - Apparently, the machines became extremely slow. Almost all tests can't pass. - One machine (worker 4) started to have the corrupt `.m2` which fails the build. - Documentation build fails time to time for an unknown reason in Jenkins machine specifically. This is disabled for now at apache#29017. - Almost all PRs are basically blocked by this instability currently. The advantages of using Github Actions: - To avoid depending on few persons who can access to the cluster. - To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce. - To control the environment more flexibly. - Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost. Note that: - The current build in Jenkins takes _more than 7 hours_. With Github actions it takes _less than 2 hours_ - We can now control the environments especially for Python easily. - The test and build look more stable than the Jenkins'. No, dev-only change. Tested at #4 Closes apache#29057 from HyukjinKwon/migrate-to-github-actions. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR proposes to enable `corssPaths` back for now to match with the build as it was. It still indeterministically doesn't run JUnit tests given my observation, and this PR basically reverts the partial fix from apache#29057. See also apache#29205 for the full context. ### Why are the changes needed? To prevent the side effects from crossPaths such as SPARK_PREPEND_CLASSES or tests that run conditionally if the test classes are present in PySpark. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested: ```bash build/sbt -Phadoop-2.7 -Phive -Phive-2.3 -Phive-thriftserver -DskipTests clean test:package ./python/run-tests --python-executable=python3 --testname="pyspark.sql.tests.test_dataframe QueryExecutionListenerTests" ``` Closes apache#29218 from HyukjinKwon/SPARK-32408-1. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>