Like other modules of Apache Griffin, measure
module is also built using Maven build tool. Building measure
module
requires Maven version 3.5+ and Java 8.
Starting from Apache Griffin 0.7, the measure
module will be (scala-spark) cross version compatible. Since both Scala
and Spark are dependencies for Apache Griffin, details of Spark-Scala cross version compatibility is mentioned below,
Spark 2.3.x | Spark 2.4.x | Spark 3.0.x | |
---|---|---|---|
Scala 2.11 | ✓ | ✓ | x |
Scala 2.12 | x | ✓ | ✓ |
Execute the below commands to build measure
with desired version of Spark and Scala,
By default, the build is compiled with Scala 2.11 and Apache Spark 2.4.x.
# For measure module with Scala 2.11 and Spark 2.4.x
mvn clean package
To change Scala or Spark version you can use the commands below,
# For measure module with Scala 2.12 and Spark 2.4.x
mvn clean package -Dscala-2.12
# For measure module with Scala 2.11 and Spark 2.3.x
mvn clean package -Dspark-2.3
# For measure module with Scala 2.12 and Spark 3.0.x
mvn clean package -Dscala-2.12 -Dspark-3.0
Note:
- Using
-Dscala-2.12
and-Dspark-2.3
option together will cause build failure due to missing dependencies as it is not cross compiled, see details here - Using
-Dspark-3.0
option without-Dscala-2.12
option will cause build failure due to missing dependencies as it is not cross compiled, see details here
Starting Spark 2.4.x, AVRO source (spark-avro
package) was migrated from com.databricks
group to org.apache.spark
.
Additionally, the older dependency does not support scala 2.12.
All builds of measure
module will contain AVRO source support by default.