[Bug]: Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version [8.12.1].Highest supported version is [7.x]. You may need to upgrade ES-Hadoop.

[Kosmas]

What happened?

Trying to run the spark dependencies job there is an error about the version of Elasticsearch and that it needs to be < 8.

It seems there was an issue #126 (closed) that has solved the issue, but using the latest image tag does not seem to be working.

On the other hand using the forked image here https://github.com/vmaleze/spark-dependencies-es8/pkgs/container/spark-dependencies-es8 works as it should

Steps to reproduce

1. Try to use the docker locally docker run -it --rm -e STORAGE=elasticsearch jaegertracing/spark-dependencies /bin/bash
2. Export necessary environment variables to connect to Elasticsearch
3. Run the command to create the dependencies java ${JAVA_OPTS} -jar $APP_HOME/jaeger-spark-dependencies-0.0.1-SNAPSHOT.jar

Expected behavior

To run the job and return successfully

24/03/25 09:28:24 INFO ElasticsearchDependenciesJob: Done, 21 dependency objects created

Relevant log output

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version [8.12.1].Highest supported version is [7.x].
You may need to upgrade ES-Hadoop.

Screenshot

No response

Additional context

No response

Jaeger backend version

v 1.54.0

SDK

OpenTelemetry Java

Pipeline

No response

Stogage backend

Elasticsearch

Operating system

Linux

Deployment model

Kubernetes

Deployment configs

No response

[yurishkuro]

but using the latest image tag does not seem to be working.

Are you loading it as latest or by specific version? latest could be stale in your environment (if you previously downloaded an older version).

[Kosmas]

@yurishkuro it is the latest one I've used, but I don't see any tags here.
Is there a specific version I could try with ?

[yurishkuro]

docker pull ghcr.io/jaegertracing/spark-dependencies/spark-dependencies@sha256:a1bfac7bae0826f878dcb62bafc8e7ff8a4df64be09393ea2dc895b32b24d0c7

[Kosmas]

Thanks @yurishkuro .

I've pulled the image tagged it locally and tried to run the job but I have the same error

Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:348)
        at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:220)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions$lzycompute(AbstractEsRDD.scala:79)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions(AbstractEsRDD.scala:78)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.getPartitions(AbstractEsRDD.scala:48)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
        at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4(Partitioner.scala:78)
        at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4$adapted(Partitioner.scala:78)
        at scala.collection.immutable.List.map(List.scala:293)
        at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:78)
        at org.apache.spark.rdd.RDD.$anonfun$groupBy$1(RDD.scala:749)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
        at org.apache.spark.rdd.RDD.groupBy(RDD.scala:749)
        at org.apache.spark.api.java.JavaRDDLike.groupBy(JavaRDDLike.scala:243)
        at org.apache.spark.api.java.JavaRDDLike.groupBy$(JavaRDDLike.scala:239)
        at org.apache.spark.api.java.AbstractJavaRDDLike.groupBy(JavaRDDLike.scala:45)
        at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:273)
        at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:249)
        at io.jaegertracing.spark.dependencies.DependenciesSparkJob.run(DependenciesSparkJob.java:54)
        at io.jaegertracing.spark.dependencies.DependenciesSparkJob.main(DependenciesSparkJob.java:40)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version [8.12.1].Highest supported version is [7.x]. You may need to upgrade ES-Hadoop.
        at org.elasticsearch.hadoop.util.EsMajorVersion.parse(EsMajorVersion.java:91)
        at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:756)
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:338)
        ... 27 more

[elMateso]

Hi, I have same problem. With latest tag is not posible run jobs and got same error. I have elastic 8.6.2. Forked image also worked me.

[LeoQuote]

Use ghcr.io/jaegertracing/spark-dependencies/spark-dependencies:latest instead of the on on docker.io , which is outdated for ages.