New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

a question with hive-es issue #601

Closed
jeromeheng opened this Issue Nov 9, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@jeromeheng

jeromeheng commented Nov 9, 2015

There is a issue with es-hive,
I have a source data table in hive

create table klogserver_item_backup(
       record_date string,
       dbname string,
       roleid string,
       rolename string,
       daoju_caozuo string,
       daoju_leixing string,
       daoju_id string,
       daoju_name string,
       backup_col string,
       daoju_count string,
       backup_col1 string,
       backup_col2 string,
       backup_col3 string,
       comment string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE;

and I create a table with es

create external table klogserver_item_es(
       record_date string,
       dbname string,
       roleid string,
       rolename string,
       daoju_caozuo string,
       daoju_leixing string,
       daoju_id string,
       daoju_name string,
       backup_col string,
       daoju_count string,
       backup_col1 string,
       backup_col2 string,
       backup_col3 string,
       comment string) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource'='index/item_es','es.index.auto.create'='true');

But I running this sentence

insert overwrite table klogserver_item_es 
  select k_item.record_date,
         k_item.dbname,
         k_item.roleid,
         k_item.rolename,
         k_item.daoju_caozuo,
         k_item.daoju_leixing,
         k_item.daoju_id,
         k_item.daoju_name,
         k_item.backup_col,
         k_item.daoju_count,
         k_item.backup_col1,
         k_item.backup_col2,
         k_item.backup_col3,
         k_item.comment from klogserver_item_backup k_item;

There is a issue :

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"record_date":"2015-11-01 11:55:16","dbname":"8024","roleid":"99470","rolename":"111","daoju_caozuo":"1","daoju_leixing":"3","daoju_id":"10","daoju_name":"1112","backup_col":"0","daoju_count":"4","backup_col1":"0","backup_col2":"0","backup_col3":"0","comment":"chapterSweepTen"}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"record_date":"2015-11-01 11:55:16","dbname":"8024","roleid":"99470","rolename":"111","daoju_caozuo":"1","daoju_leixing":"3","daoju_id":"10","daoju_name":"1112","backup_col":"0","daoju_count":"4","backup_col1":"0","backup_col2":"0","backup_col3":"0","comment":"chapterSweepTen"}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 2.0.0
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:577)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:675)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
    ... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 2.0.0
    at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:249)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:622)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:566)
    ... 16 more
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 2.0.0
    at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:172)
    at org.elasticsearch.hadoop.hive.HiveUtils.init(HiveUtils.java:142)
    at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:93)
    at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:42)
    at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:261)
    at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:246)
    ... 18 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-0: Map: 2   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

how to solve the problem ?
hive 1.2.0
hadoop 2.6.0
es 1.44
es-hadoop elasticsearch-hadoop-2.1.2.jar

@costin

This comment has been minimized.

Show comment
Hide comment
@costin

costin Nov 11, 2015

Member

You are using Elasticsearch 2.0 which is not supported by es-hadoop-2.1.2. You need to use es-hadoop-2.2.0 instead (currently in beta1).

Cheers,

Member

costin commented Nov 11, 2015

You are using Elasticsearch 2.0 which is not supported by es-hadoop-2.1.2. You need to use es-hadoop-2.2.0 instead (currently in beta1).

Cheers,

@costin costin closed this Nov 11, 2015

@costin costin added the invalid label Nov 11, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment