Skip to content

Conversation

@RussellSpitzer
Copy link
Member

Due to an issue with the build.gradle both the spark test and test31 modules were both running with Spark 3.1. Removing the inter dependency fixes the issue and both grade tasks now run with the correct respective spark versions.

@cwsteinbach

@github-actions github-actions bot added the build label Jul 26, 2021
@RussellSpitzer RussellSpitzer changed the title Build: Remove Dependency on Test31 for Spark Tests Build: Run tests against Spark 3.0 and Spark 3.1 Jul 26, 2021
@RussellSpitzer
Copy link
Member Author

@rdblue - @karuppayya noticed this when we were debugging the issue for #2783

@RussellSpitzer
Copy link
Member Author

➜  iceberg-master git:(FixBuildAgainstSparkVersions) ✗ g check --dry-run | grep testSpark31
:iceberg-spark3:testSpark31 SKIPPED
:iceberg-spark3-extensions:testSpark31 SKIPPED
➜  iceberg-master git:(FixBuildAgainstSparkVersions) ✗ g :iceberg-spark3:dependencies --configuration compileOnly | grep spark-hive
\--- org.apache.spark:spark-hive_2.12:3.0.3
➜  iceberg-master git:(FixBuildAgainstSparkVersions) ✗ g :iceberg-spark3:dependencies --configuration testRuntimeClasspath | grep spark-hive
+--- org.apache.spark:spark-hive_2.12:3.0.3


// Only compiles tests
➜  iceberg-master git:(FixBuildAgainstSparkVersions) ✗ g :iceberg-spark3:dependencies --configuration spark31CompileClasspath | grep spark-hive
+--- org.apache.spark:spark-hive_2.12:3.0.3 -> 3.1.1
+--- org.apache.spark:spark-hive_2.12:3.1.1 (*)
➜  iceberg-master git:(FixBuildAgainstSparkVersions) ✗ g :iceberg-spark3:dependencies --configuration spark31RuntimeClasspath | grep spark-hive
+--- org.apache.spark:spark-hive_2.12:3.0.3 -> 3.1.1
+--- org.apache.spark:spark-hive_2.12:3.1.1 (*)

With this dependency in place, Tests were running with 31 regardless of whether
it was test31 or not.
@RussellSpitzer RussellSpitzer force-pushed the FixBuildAgainstSparkVersions branch from d57f353 to 9fd1a45 Compare July 26, 2021 23:41
@RussellSpitzer
Copy link
Member Author

Had to make an additional change to trigger both sets of tests to run during gradle check command, rerunning suite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants