New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES 2.0 SSL problem #608

Closed
pricecarl opened this Issue Nov 13, 2015 · 4 comments

Comments

Projects
None yet
2 participants
@pricecarl

pricecarl commented Nov 13, 2015

Hey,

I'm having trouble writing data from hive to ES 2.0 I'm using the elasticsearch-hadoop-hive-2.2.0-beta1.jar file but am getting the below error:

Caused by: org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:363)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:367)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:121)
at org.elasticsearch.hadoop.rest.RestClient.getHttpDataNodes(RestClient.java:336)
at org.elasticsearch.hadoop.rest.InitializationUtils.filterNonDataNodesIfNeeded(InitializationUtils.java:121)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:381)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)  
... 9 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:430)
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:104)
... 31 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
... 49 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
    ... 55 more

Neither my Hadoop or ES 2.0 cluster uses SSL so I don't know why its asking for certs. According to the docs SSL is defaulted to false but I have put it in my table properties again in case:

...
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES ('es.resource' = 'proxylog-2015-11-02/event',
'es.index.auto.create' = 'true',
'es.nodes' = 'calhdb08.cyber.lab',
'es.port' = '9200',
'es.field.read.empty.as.null' ='true',
'es.net.ssl' = 'false',
'es.batch.size.entries' = '10000',
'es.batch.size.bytes' = '128mb',
'es.mapping.names' = 'Time
...

Any ideas would be greatly appreciated.

I have also raised this with ES support.

@costin

This comment has been minimized.

Show comment
Hide comment
@costin

costin Nov 14, 2015

Member

Something is definitely is using SSL otherwise this won't show in the stack trace. Based on the strack trace is looks like the initial call to ES is successful but the subsequent call ends being wrapped through SSL. Can you please enable TRACE logging on the rest package and upload the gist somewhere?

Additionally, please double check your configuration - the SSL won't appear out of nowhere; maybe you have a transparent proxy defined or maybe the ES nodes communicate between each other using SSL. Does curl works without SSL? Since ES is accessible as REST, you should be able to simply point the browser to it as well.
Anything special in the ES logs?

Member

costin commented Nov 14, 2015

Something is definitely is using SSL otherwise this won't show in the stack trace. Based on the strack trace is looks like the initial call to ES is successful but the subsequent call ends being wrapped through SSL. Can you please enable TRACE logging on the rest package and upload the gist somewhere?

Additionally, please double check your configuration - the SSL won't appear out of nowhere; maybe you have a transparent proxy defined or maybe the ES nodes communicate between each other using SSL. Does curl works without SSL? Since ES is accessible as REST, you should be able to simply point the browser to it as well.
Anything special in the ES logs?

@pricecarl

This comment has been minimized.

Show comment
Hide comment
@pricecarl

pricecarl Nov 18, 2015

Hi Costin,

Forgive me I'm not great with the whole hadoop setup and I'm not sure how to enable logging.

I was however able to successfully run curl commands against the cluster:

curl -XGET http://calhdb08.cyber.lab:9200
{
  "name" : "calhdb08-es-ClientNode",
  "cluster_name" : "ucp-incubator-proxy",
  "version" : {
"number" : "2.0.0",
"build_hash" : "de54438d6af8f9340d50c5c786151783ce7d6be5",
"build_timestamp" : "2015-10-22T08:09:48Z",
"build_snapshot" : false,
"lucene_version" : "5.2.1"
  },
  "tagline" : "You Know, for Search"
 }

and:

curl -XPUT http://calhdb08.cyber.lab:9200/carltest2_index/carltest_type/1 -d '{"Hello" : "world"}'

{"_index":"carltest2_index","_type":"carltest_type","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Also I've been doing some digging and found this option:

security.manager.enabled: false

Which I put into my elasticsearch.yml file, now I get this error:

Task with the most failures(4): 
-----
Task ID:
  task_1446551850158_0203_m_000001

URL:
  http://ucp-namenode-01.ucp.cyber.lab:8088/taskdetails.jsp?jobid=job_1446551850158_0203&tipid=task_1446551850158_0203_m_000001
-----
 Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 

<data removed>

at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
<data removed>
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[calhdb09.cyber.lab/192.168.24.170:9200]] 
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.executeNotFoundAllowed(RestClient.java:391)
at org.elasticsearch.hadoop.rest.RestClient.exists(RestClient.java:467)
at org.elasticsearch.hadoop.rest.RestClient.touch(RestClient.java:473)
at org.elasticsearch.hadoop.rest.RestRepository.touch(RestRepository.java:473)
at org.elasticsearch.hadoop.rest.RestService.initSingleIndex(RestService.java:411)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:399)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more

Might be barking completely up the wrong tree here though, what do you think?

pricecarl commented Nov 18, 2015

Hi Costin,

Forgive me I'm not great with the whole hadoop setup and I'm not sure how to enable logging.

I was however able to successfully run curl commands against the cluster:

curl -XGET http://calhdb08.cyber.lab:9200
{
  "name" : "calhdb08-es-ClientNode",
  "cluster_name" : "ucp-incubator-proxy",
  "version" : {
"number" : "2.0.0",
"build_hash" : "de54438d6af8f9340d50c5c786151783ce7d6be5",
"build_timestamp" : "2015-10-22T08:09:48Z",
"build_snapshot" : false,
"lucene_version" : "5.2.1"
  },
  "tagline" : "You Know, for Search"
 }

and:

curl -XPUT http://calhdb08.cyber.lab:9200/carltest2_index/carltest_type/1 -d '{"Hello" : "world"}'

{"_index":"carltest2_index","_type":"carltest_type","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Also I've been doing some digging and found this option:

security.manager.enabled: false

Which I put into my elasticsearch.yml file, now I get this error:

Task with the most failures(4): 
-----
Task ID:
  task_1446551850158_0203_m_000001

URL:
  http://ucp-namenode-01.ucp.cyber.lab:8088/taskdetails.jsp?jobid=job_1446551850158_0203&tipid=task_1446551850158_0203_m_000001
-----
 Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 

<data removed>

at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
<data removed>
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[calhdb09.cyber.lab/192.168.24.170:9200]] 
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:383)
at org.elasticsearch.hadoop.rest.RestClient.executeNotFoundAllowed(RestClient.java:391)
at org.elasticsearch.hadoop.rest.RestClient.exists(RestClient.java:467)
at org.elasticsearch.hadoop.rest.RestClient.touch(RestClient.java:473)
at org.elasticsearch.hadoop.rest.RestRepository.touch(RestRepository.java:473)
at org.elasticsearch.hadoop.rest.RestService.initSingleIndex(RestService.java:411)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:399)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more

Might be barking completely up the wrong tree here though, what do you think?

@pricecarl

This comment has been minimized.

Show comment
Hide comment
@pricecarl

pricecarl Nov 23, 2015

Hi Costin,

I changed a setting in the elasticsearch.yml file to use the nodes IP instead of its hostname for publishing and this has fixed the issue.

I can't seem to replicate the SSL issue though but I definitely do not use SSL anywhere on this cluster, so a bit of a strange one.

pricecarl commented Nov 23, 2015

Hi Costin,

I changed a setting in the elasticsearch.yml file to use the nodes IP instead of its hostname for publishing and this has fixed the issue.

I can't seem to replicate the SSL issue though but I definitely do not use SSL anywhere on this cluster, so a bit of a strange one.

@costin costin added v2.2.0 and removed v2.2.0-rc1 labels Jan 8, 2016

@costin

This comment has been minimized.

Show comment
Hide comment
@costin

costin Jan 15, 2016

Member

Closing this one since it seems to be fixed. If that's not the case, please reopen it.

Member

costin commented Jan 15, 2016

Closing this one since it seems to be fixed. If that's not the case, please reopen it.

@costin costin closed this Jan 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment