Skip to content

Commit

Permalink
[SPARK-33079][TESTS] Replace the existing Maven job for Scala 2.13 in…
Browse files Browse the repository at this point in the history
… Github Actions with SBT job

### What changes were proposed in this pull request?

SPARK-32926 added a build test to GitHub Action for Scala 2.13 but it's only with Maven.
As SPARK-32873 reported, some compilation error happens only with SBT so I think we need to add another build test to GitHub Action for SBT.
Unfortunately, we don't have abundant resources for GitHub Actions so instead of just adding the new SBT job, let's replace the existing Maven job with the new SBT job for Scala 2.13.

### Why are the changes needed?

To ensure build test passes even with SBT for Scala 2.13.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

GitHub Actions' job.

Closes #29958 from sarutak/add-sbt-job-for-scala-2.13.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
sarutak authored and HyukjinKwon committed Oct 15, 2020
1 parent e85ed8a commit 513b6f5
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 11 deletions.
16 changes: 6 additions & 10 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -301,22 +301,18 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
- name: Cache Maven local repository
- name: Cache Ivy local repository
uses: actions/cache@v2
with:
path: ~/.m2/repository
key: scala-213-maven-${{ hashFiles('**/pom.xml') }}
path: ~/.ivy2/cache
key: scala-213-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
scala-213-maven-
scala-213-ivy-
- name: Install Java 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Build with Maven
- name: Build with SBT
run: |
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
export MAVEN_CLI_OPTS="--no-transfer-progress"
mkdir -p ~/.m2
./dev/change-scala-version.sh 2.13
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 -Pscala-2.13 install
rm -rf ~/.m2/repository/org/apache/spark
./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Djava.version=11 -Pscala-2.13 compile test:compile
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ class KinesisBackedBlockRDD[T: ClassTag](
require(_blockIds.length == arrayOfseqNumberRanges.length,
"Number of blockIds is not equal to the number of sequence number ranges")

override def isValid(): Boolean = true
override def isValid: Boolean = true

override def getPartitions: Array[Partition] = {
Array.tabulate(_blockIds.length) { i =>
Expand Down

0 comments on commit 513b6f5

Please sign in to comment.