Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A null pointer exception is thrown when a node in a cluster does not have a defined "http_address" #99

Closed
dreverri opened this issue Oct 15, 2013 · 3 comments
Closed

A null pointer exception is thrown when a node in a cluster does not have a defined "http_address" #99

dreverri opened this issue Oct 15, 2013 · 3 comments
Labels
:Rest v1.3.0.M2

Comments

@dreverri
Copy link

Version: elasticsearch-hadoop-1.3.0.M1

The absence of the http_address property causes the following stack trace:

java.lang.NullPointerException
    at org.elasticsearch.hadoop.rest.dto.Node.<init>(Node.java:30)
    at org.elasticsearch.hadoop.rest.RestClient.getNodes(RestClient.java:139)
    at org.elasticsearch.hadoop.rest.BufferedRestClient.getTargetShards(BufferedRestClient.java:224)
    at org.elasticsearch.hadoop.mr.ESInputFormat.getSplits(ESInputFormat.java:354)
    at org.elasticsearch.hadoop.mr.ESInputFormat.getSplits(ESInputFormat.java:335)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
    at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)

Here is the output of http://localhost:9200/_nodes for the cluster that caused the issue:

{
    "ok": true,
    "cluster_name": "elasticsearch",
    "nodes": {
        "0SYpLdO2TNmu8UB7JVHGRg": {
            "hostname": "new-host.home",
            "http_address": "inet[/127.0.0.1:9200]",
            "name": "Demiurge",
            "transport_address": "inet[/127.0.0.1:9300]",
            "version": "0.90.0"
        },
        "SJ_OybHbTL-cgkAIbn7Aqw": {
            "attributes": {
                "data": "false",
                "master": "false"
            },
            "hostname": "new-host.home",
            "name": "Williams, Eric",
            "transport_address": "inet[/192.168.1.2:9300]",
            "version": "0.90.3"
        }
    }
}
@costin costin added v1.3.0.M2 and removed mr labels Feb 6, 2014
@costin
Copy link
Member

costin commented Feb 6, 2014

Hi,

This should be fixed in 1.3 M2 - if that is not the case, please let me know. Cheers!

@costin costin closed this as completed Feb 6, 2014
@ccrivelli
Copy link

Hi Costin,
I'm encountering the same problem exporting from hive/hdfs to elasticsearch.

This is the hive table created for the export process:

ADD JAR /opt/aux_jars/elasticsearch-hadoop-2.1.0.BUILD-20140528.012021-1.jar;
CREATE EXTERNAL TABLE IF NOT EXISTS 999_test_tweets
(
id BIGINT,
datehour STRING
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
LOCATION '/data_flow/elastic'
TBLPROPERTIES('es.resource' = 'kibana-int/tweets', 'es.nodes' = 'elasticsearch_node:9200', 'es.index.read.missing.as.empty' = 'yes', 'es.index.auto.create' = 'yes', 'es.field.read.empty.as.null' = 'yes', 'es.mapping.names' = 'datehour:@timestamp')

And this is the Export query:

ADD JAR /opt/aux_jars/elasticsearch-hadoop-2.1.0.BUILD-20140528.012021-1.jar;
ADD JAR /opt/aux_jars/hive-serdes-1.0-SNAPSHOT.jar;

INSERT OVERWRITE TABLE 999_test_tweets
SELECT t.id,
CAST(CONCAT(SUBSTR(t.datehour,1,4),'/',SUBSTR(t.datehour,5,2),'/',SUBSTR(t.datehour,7,2),' ',SUBSTR(t.datehour,9,2),':00:00') as STRING)
FROM tweets_test t;

Around a couple of days ago the export process was working fine, but this morning I realized It stopped to work and started to get this error:

Error while processing row {"id":469039897471647744,"datehour":"2014052119"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: java.lang.NullPointerException
at org.elasticsearch.hadoop.serialization.dto.Node.(Node.java:33)
at org.elasticsearch.hadoop.rest.RestClient.getNodes(RestClient.java:245)
at org.elasticsearch.hadoop.rest.RestRepository.getWriteTargetPrimaryShards(RestRepository.java:240)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.initSingleIndex(EsOutputFormat.java:218)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:201)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more
Container killed by the ApplicationMaster.

elastic search is "version":"1.1.1"

Can you please help me to find the solution or at least address me in the right direction?

Thanks.

@costin
Copy link
Member

costin commented May 29, 2014

Probably (some of) your nodes don't have http enabled. I've created a new issue to track this down since the current one refers to a different version.

@costin costin mentioned this issue May 29, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Rest v1.3.0.M2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dreverri @costin @ccrivelli