version mismatch exception.

[realrnjs]

What kind an issue is this?

• Bug report. If you’ve found a bug, please provide a code snippet or test to reproduce it below.
  The easier it is to track down the bug, the faster it is solved.
• Feature Request. Start by telling us what problem you’re trying to solve.
  Often a solution already exists! Don’t send pull requests to implement new features without
  first getting our support. Sometimes we leave features out on purpose to keep the project small.

Issue description

Using the cloudera library results in a version mismatch exception.

Steps to reproduce

Code:

val query =
      df.groupBy("topic", "partition")
        .max("offset")
        .writeStream
        .format("es")
        .outputMode(OutputMode.Complete())
        .trigger(Trigger.ProcessingTime(1, TimeUnit.SECONDS))
        .option(ConfigurationOptions.ES_SPARK_DATAFRAME_WRITE_NULL_VALUES,
                "true")
        .option("checkpointLocation", checkpointLocation)
        .start(s"$topic/offset")

Strack trace:

Spark version mismatch. Expected at least Spark version [2.2.0] but found [2.2.0.cloudera1]. Spark Structured Streaming is a feature that is only supported on Spark [2.2.0] and up for this version of ES-Hadoop/Spark.
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Spark version mismatch. Expected at least Spark version [2.2.0] but found [2.2.0.cloudera1]. Spark Structured Streaming is a feature that is only supported on Spark [2.2.0] and up for this version of ES-Hadoop/Spark.
	at org.elasticsearch.spark.sql.streaming.StructuredStreamingVersionLock$.checkCompatibility(StructuredStreamingVersionLock.scala:31)
	at org.elasticsearch.spark.sql.DefaultSource.createSink(DefaultSource.scala:119)
	at org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:272)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:286)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:222)

Version Info

OS: : Ubuntu 16.4
JVM : 1.8
Hadoop/Spark: 2.6/2.2.0
ES-Hadoop : 6.0.0-rc2
ES : 5.6.2

Feature description

[jbaiera]

Thanks for the report @realrnjs. This was a quick fix and a good one to find while 6.x is still in RC.

[realrnjs]

Thank you.