[PIO-171] Drop Scala 2.10 and Spark 1.6 support #471
Conversation
takezoe
commented
Sep 22, 2018
•
takezoe
commented
- Build configuration update
- Documentation update
takezoe
changed the title
| @@ -17,24 +17,7 @@ | |||
| import PIOBuild._ | |||
|
|
|||
| lazy val scalaSparkDepsVersion = Map( | |||
There was a problem hiding this comment.
Maybe we can get rid of this Map as well?
There was a problem hiding this comment.
Yes, but we will add Scala 2.12 support very soon. So keeping this Map is better I think.
| * Apache Hadoop 2.6.5 (optional, required only if YARN and HDFS are needed) | ||
| * Apache Spark 1.6.3 for Hadoop 2.6 | ||
| * Apache Hadoop 2.7.3 (optional, required only if YARN and HDFS are needed) | ||
| * Apache Spark 2.1.1 for Hadoop 2.7 |
There was a problem hiding this comment.
I wonder if we should put Hadoop 2.6.x/2.7.x, and Spark 2.0+ here. We still run CI on Spark 2.0, and the Hadoop requirement comes mostly from Spark.
There was a problem hiding this comment.
I coundn't find Spark 2.0 pattern in CI and default Hadoop version for PredictionIO build is now Hadoop 2.7.3. But in build.sbt, Spark 2.0 is still supported and we can specify Hadoop version as 2.6.x. What is minimum supported versions in this case...?
There was a problem hiding this comment.
How about we try adding an entry to Travis to build against Spark 2.0.2 and Hadoop 2.6.5? If that passes, we could still support that. I do notice that Spark 2.0.x has not received updates for almost 2 years now.
|
We should also update |
|
I added Spark 2.0.2 / Hadoop 2.6.5 test patterns and updated Spark version in |
|
@takezoe Looks great. I have no further comments. Thank you! |
|
Just stumbled upon https://github.com/apache/predictionio/blob/develop/data/src/main/spark-1/org/apache/predictionio/data/SparkVersionDependent.scala. We probably should clean this up as well. |
|
@dszeto Nice catch! I removed the file. |
|
Merging. Thank you! |
