7.17.11 failed backwards compatibility for 5.5.3

[sj-ganwh]

What kind an issue is this?

• Bug report. If you’ve found a bug, please provide a code snippet or test to reproduce it below.
  The easier it is to track down the bug, the faster it is solved.
• Feature Request. Start by telling us what problem you’re trying to solve.
  Often a solution already exists! Don’t send pull requests to implement new features without
  first getting our support. Sometimes we leave features out on purpose to keep the project small.

Issue description

Due to CVE-2023-46674, we need to update version to Elasticsearch-hadoop >= 7.17.11 or >= 8.9.0. So we plan to update to 7.17.11.
According to https://github.com/elastic/elasticsearch-hadoop quote ES-Hadoop 6.x and higher are compatible with Elasticsearch 1.X, 2.X, 5.X, and 6.X, so ES-Hadoop 7.x and higher are compatible with Elasticsearch 1.X, 2.X, 5.X, 6.X and 7.X.
But when we replace elasticsearch-hadoop-5.5.3.jar to elasticsearch-hadoop-7.17.11.jar, it fails.

Steps to reproduce

Code:

-- add jar hdfs:///lib/jdbc/elasticsearch-hadoop-5.5.3.jar;
add jar hdfs:///lib/jdbc/elasticsearch-hadoop-7.17.11.jar;

create table default.tmp_es (
    id bigint,
    name string,
    update_time timestamp
)
stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
tblproperties (
    "es.nodes"="172.26.1.1:9200",
    "es.net.http.header.Authorization"="Basic xxx",
    "es.nodes.wan.only"="true",
    "es.nodes.discovery"="false",
    "es.http.retries"="10",
    -- index/type
    "es.resource"="tmp_es/doc",
    "es.mapping.id"="id",
    -- dst:src
    "es.mapping.names"="update_time:updateTime"
);

select * from default.tmp_es;

Strack trace:

Failed with exception java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

Version Info

OS: : CentOS 7
JVM : jdk8
Hadoop/Spark: hadoop 2.7
Hive: 2.1.1
ES-Hadoop : 7.17.11
ES : 5.5.3

[sj-ganwh]

Tested, elasticsearch-hadoop-7.1.1.jar is compatible with Elasticsearch 5.5.3.

[sj-ganwh]

CVE-2023-46674 further reading: https://discuss.elastic.co/t/elasticsearch-hadoop-7-17-11-8-9-0-security-update-esa-2023-28/348663

[sj-ganwh]

I changed major.before(EsMajorVersion.V_6_X to major.before(EsMajorVersion.V_5_X in
archongum@52d5505 and rebuild from it and it worked... These are the artifacts: https://github.com/sj-ganwh/elasticsearch-hadoop/actions/runs/8014107473
In my case, I just using Hive external table just to read from ES 5.5.3, this kind of 'force fix' has any side effects?

[jbaiera]

In 7.14 we added a validation check to the library to ensure that ES-Hadoop was contacting an Elasticsearch distribution and not something else during it's setup process. #1696 The header is only present in more recent versions of Elasticsearch. The connector technically could still work for older versions of Elasticsearch but without the header present it will reject connecting further beyond the initial handshake. The header should be present in Elasticsearch before upgrading ES-Hadoop to 7.14.