Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Target Spark 1.5.2 as default Spark version #911
Moving to Spark 1.6 would have the benefit that you can migrate the Adam library to use the DataSet API.
I am not sure if it's worth moving to the DataFrame API in Spark 1.5.2. Since the DataFrame API is incompatible with some transformations /code in this library (e.g.
Spark 1.5.2 is now widely used but I guess people will move fast to Spark 1.6 if the DataSet API turns out to work / be stable?
Since there is no currently large (production) user base for the Adam library it's easy to move fast to Spark 1.6 / the DataSet API? (and drop support for older Spark versions?)