New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot initialize SSL - "Cannot open stream" exception by EsHadoop connector for spark after upgrading to version 2.2 #1018

Closed
yosid16 opened this Issue Jul 19, 2017 · 8 comments

Comments

Projects
None yet
3 participants
@yosid16

yosid16 commented Jul 19, 2017

Issue description

After upgrading local version of spark to 2.2, EsHadoop connector fails to connector to remote elastic cluster with the following error:

Steps to reproduce

Any read/write operation fails, for example(replace the path/host prior to executing this):

    val conf = new SparkConf().set("spark.master", "local[2]")
    val sparkSession = SparkSession.builder().config(conf).getOrCreate()
    import spark.implicits._
    val df: DataFrame = Seq("somevalue").toDF("mycol")
    val opts = Map(
      "es.index.auto.create" -> "true",
      "es.read.metadata" -> "true",
      "spark.serializer" -> "org.apache.spark.serializer.KryoSerializer",
      "es.net.http.auth.user" -> "admin",
      "es.net.http.auth.pass" -> "somepass",
      "es.net.ssl" -> "true",
      "es.net.ssl.cert.allow.self.signed" -> "true",
      "es.net.ssl.truststore.location" -> "//some/path",
      "es.net.ssl.truststore.pass" -> "ibicloud",
      "es.nodes" -> "somehost",
      "es.port" -> "9997"
    )

    df.saveToEs("someindex",opts)

Strack trace:

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:283)
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:572)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:96)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:96)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:168)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:153)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:122)
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:478)
	at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:112)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:461)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:425)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:429)
	at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:155)
	at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:635)
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:276)
	... 10 more
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:180)
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:185)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:178)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadTrustManagers(SSLSocketFactory.java:204)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:166)
	... 25 more
17/07/19 19:52:16 WARN TaskSetManager: Lost task 0.0 in stage 13.0 (TID 815, localhost, executor driver): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:283)
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:572)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:96)
	at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:96)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:168)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:153)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:122)
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:478)
	at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:112)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:461)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:425)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:429)
	at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:155)
	at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:635)
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:276)
	... 10 more
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:180)
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:185)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:178)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadTrustManagers(SSLSocketFactory.java:204)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:166)
	... 25 more

Version Info

I've first experienced this with ES-Hadoop version 5.4.1. Tried to upgrade it to all version until latest (6.0.0-alpha2) and it didn't resolve the issue.

OS: : Windows
JVM : 1.8
Hadoop/Spark: 2.2
ES-Hadoop : 5.4.1 - 6.0.0-alpha2
ES : 5.4.0

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Jul 19, 2017

Contributor

@yosid16 Is this the complete stack trace? Is there anything else in the logs for the job that you can post here?

Contributor

jbaiera commented Jul 19, 2017

@yosid16 Is this the complete stack trace? Is there anything else in the logs for the job that you can post here?

@yosid16

This comment has been minimized.

Show comment
Hide comment
@yosid16

yosid16 Jul 19, 2017

Updated the stack trace - now it's complete. No errors except that one. There are no errors except that one in the log.

yosid16 commented Jul 19, 2017

Updated the stack trace - now it's complete. No errors except that one. There are no errors except that one in the log.

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Jul 19, 2017

Contributor

@yosid16 This seems to be a completely different stack trace now. Instead of the SSL error, it looks like it was able to open the connection and received back a 503. Can you post the entire set of logs here?

Contributor

jbaiera commented Jul 19, 2017

@yosid16 This seems to be a completely different stack trace now. Instead of the SSL error, it looks like it was able to open the connection and received back a 503. Can you post the entire set of logs here?

@yosid16

This comment has been minimized.

Show comment
Hide comment
@yosid16

yosid16 Jul 19, 2017

Sure (removed host names, and some file paths)
spark.zip

yosid16 commented Jul 19, 2017

Sure (removed host names, and some file paths)
spark.zip

@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Jul 19, 2017

Contributor

Thank you for adding the full logs. This is the line here that jumped out for me:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:180)
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:185)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:178)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadTrustManagers(SSLSocketFactory.java:204)
  1. That's a horrid error message which is incredibly misleading. I'll be committing a fix to make that message less confusing.
  2. The connector cannot find or open your truststore file. Once again, the root reason for why it cannot is lost because we swallow the exception instead of passing it as the cause for the exception. Another thing I'll see about fixing.
  3. I'm assuming that you stripped the file name out, which is why it displays file:/// instead of any actual truststore file. The truststore file needs to exist on every node for the connector to be able to function.

I'm going to push some commits to master to clean up those error messages. I saw that you're testing this on the alpha release as well. If you checkout the next snapshot build hopefully it will illuminate the problem a bit better. The last snapshot build was three hours ago as of this post, so some time tomorrow about this time it should be available.

Contributor

jbaiera commented Jul 19, 2017

Thank you for adding the full logs. This is the line here that jumped out for me:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot open stream (inlined queries need to be marked as such through `?` and `{}`) for resource file://
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:180)
	at org.elasticsearch.hadoop.util.IOUtils.open(IOUtils.java:185)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:178)
	at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadTrustManagers(SSLSocketFactory.java:204)
  1. That's a horrid error message which is incredibly misleading. I'll be committing a fix to make that message less confusing.
  2. The connector cannot find or open your truststore file. Once again, the root reason for why it cannot is lost because we swallow the exception instead of passing it as the cause for the exception. Another thing I'll see about fixing.
  3. I'm assuming that you stripped the file name out, which is why it displays file:/// instead of any actual truststore file. The truststore file needs to exist on every node for the connector to be able to function.

I'm going to push some commits to master to clean up those error messages. I saw that you're testing this on the alpha release as well. If you checkout the next snapshot build hopefully it will illuminate the problem a bit better. The last snapshot build was three hours ago as of this post, so some time tomorrow about this time it should be available.

@yosid16

This comment has been minimized.

Show comment
Hide comment
@yosid16

yosid16 Jul 19, 2017

Thanks for checking it. This runs locally, I have access to the truststore path which does exist (when spark runs with version 2.1 it enables to access it). I'll try again tomorrow when we have more information about the cause.

yosid16 commented Jul 19, 2017

Thanks for checking it. This runs locally, I have access to the truststore path which does exist (when spark runs with version 2.1 it enables to access it). I'll try again tomorrow when we have more information about the cause.

jbaiera added a commit that referenced this issue Jul 19, 2017

Cleaning up some poor Exception handling in IOUtils.
This throws an unhelpful error when trying to load a resource. First, it
assumes that the resource is being used for the connector's
user-specified-pushdown query. Second, it swallows the root reason for
not being able to open the resource for reading. This is confusing if
you are not loading a file that has anything to do with the pushdown
query, in addition to not having any info about why the connector cannot
retrieve the given resource.

relates #1018
@jbaiera

This comment has been minimized.

Show comment
Hide comment
@jbaiera

jbaiera Aug 25, 2017

Contributor

I'm going to close this as this issue has been inactive for a while. If anything comes up, please let us know and we can re-open the issue if need be.

Contributor

jbaiera commented Aug 25, 2017

I'm going to close this as this issue has been inactive for a while. If anything comes up, please let us know and we can re-open the issue if need be.

@jbaiera jbaiera closed this Aug 25, 2017

@jbaiera jbaiera added question and removed feedback-requested labels Aug 25, 2017

@dmarkhas

This comment has been minimized.

Show comment
Hide comment
@dmarkhas

dmarkhas Jan 27, 2018

Contributor

So, the problem is actually not with the connector itself (although it can provide a workaround) but in the core spark libraries.
Since IOUtils.open() does:
new URL(resource).openStream()

It relies on the URL stream handler factory, which is being set by Spark to org.apache.hadoop.fs.FsUrlStreamHandlerFactory inside SharedState.scala

It is possible that the behavior of the hadoop RawLocalFileSystem changed between 2.1 and 2.2.0, but either way - UNC paths for loading the truststore (such as in the original post) no longer work, and they did in 2.1.
The connector could provide a workaround by recognize the UNC path and loading the file directly.
So for a path like: "\\shared\folder\truststore.jks" something along these lines would work in IOUtils.open():

       if (resource.startsWith("\\"))
        {
            return new FileInputStream(new File(resource));
        }
        // no prefix means classpath
        else if (!resource.contains(":")) {
            return loader.getResourceAsStream(resource);
        }
      return new URL(resource).openStream();
Contributor

dmarkhas commented Jan 27, 2018

So, the problem is actually not with the connector itself (although it can provide a workaround) but in the core spark libraries.
Since IOUtils.open() does:
new URL(resource).openStream()

It relies on the URL stream handler factory, which is being set by Spark to org.apache.hadoop.fs.FsUrlStreamHandlerFactory inside SharedState.scala

It is possible that the behavior of the hadoop RawLocalFileSystem changed between 2.1 and 2.2.0, but either way - UNC paths for loading the truststore (such as in the original post) no longer work, and they did in 2.1.
The connector could provide a workaround by recognize the UNC path and loading the file directly.
So for a path like: "\\shared\folder\truststore.jks" something along these lines would work in IOUtils.open():

       if (resource.startsWith("\\"))
        {
            return new FileInputStream(new File(resource));
        }
        // no prefix means classpath
        else if (!resource.contains(":")) {
            return loader.getResourceAsStream(resource);
        }
      return new URL(resource).openStream();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment