Elastic - spark : deprecated search_type=scan showing up in post url from spark 2.4 and returning 0 hits

[john-marshal]

What kind an issue is this?

Am trying to fetch data from elastic server into spark 2.4 environment using "org.elasticsearch.spark.sql" connector, but even though i got all the configs right, there is no data received on the spark engine

Using the below config to connect to Elastic from spark

Issue description

On analyzing with fiddler found that the request created had a search_type=scan param in the query string, which i guess is no longer supported

http://NodeURL/index/type/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local

On opening this on the browser didn't receive any data, but the same query without the search_type param worked as expected.

Am wondering when the above param is no longer supported, why is it being appended in the request url by the elasticsearch-spark connector

Version Info

spark - 2.4.0
scala - 2.11
elasticsearch-spark-20_2.11 (6.6.0)
OS: : Win 10
JVM : jdk1.8.0_181

Can someone please help me figure out a way to fetch data

[jbaiera]

Which version of Elasticsearch are you running against?

[jbaiera]

The only reason the search_type parameter gets added to the request is if it is running on a version earlier than 5.0, and there is currently an existing issue where more recent versions of ES-Hadoop reading from 2.x versions of Elasticsearch are not returning results. See #1277

[jbaiera]

Closing this as there has been no further feedback