Spark 2.1 support

[acchen97]

This issue tracks adding support for Spark 2.1.

http://spark.apache.org/releases/spark-release-2-1-0.html

[jbaiera]

The upgrade process didn't run into any issues. This should be easy to get in for 5.2.

[pfcoperez]

@acchen97 @jbaiera I've performed some simple examples and it seems to work nicely with Spark 2.1 using local mode.

Is it the case that this task cover the fact that it has to be thoroughly checked before being considered as officially supported? just that a new version needs to be released addressing Spark 2.1 dependencies (as it currently fails because serialization signatures in standalone mode)?

If not, what cases are failing in Spark 2.1?

Thanks

[acchen97]

@pfcoperez thanks for testing it. Yes, it's predominantly making sure everything works before we consider Spark 2.1 formally supported.

Let us know if anything else pops up for you.

[jbaiera]

Released in 5.2.0

[cjuexuan]

hi,I found last elasticsearch-spark_2.11 version is 5.0.0-alpha4,and use spark_1.6,and in elasticsearch-hadoop-5.2.0 ,scala version is 2.10,so I think we need a version adaptation spark_2.11 and scala_2.11

[jbaiera]

@cjuexuan Please see the Spark section of the installation guide in the documentation. It outlines the correct artifacts for usage when working with the different versions of Spark and Scala.

[cjuexuan]

@jbaiera ，thanks