Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with inserting/parsing timestamps from hive to elasticsearch. #259

Closed
snarla opened this issue Sep 2, 2014 · 1 comment
Closed

Issue with inserting/parsing timestamps from hive to elasticsearch. #259

snarla opened this issue Sep 2, 2014 · 1 comment
Labels
bug enhancement :Hive v2.0.2 v2.1.0.Beta2

Comments

@snarla
Copy link

snarla commented Sep 2, 2014

I am facing issues with inserting "date_time" type from Hive to Elasticseach. This is the table definition I used:
create external table sc1m4 (
dim1 string,
ms2 double,
hiveid string,
ts string
) stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'scenarios/sc1m4','es.nodes'='headnode0',
'es.mapping.ttl'='5m',
'es.mapping.id'='hiveid',
'es.mapping.timestamp'='ts');

The value for ts I am trying to insert: "2014-08-26T12:44:00.000"
The exception I see:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{},"value":{"_col0":"dim_valueY","_col1":89.25,"_col2":"dim_valueY2014-08-2612:43:00","_col3":"2014-08-26T12:4
4:00.000"}}
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{},"value":{"_col0":"dim_valueY","_col1":89.25,"_col2":"dim_valueY2014-08-2612:43:00","_col3":"2014-08-26T12:44:00.000"}}
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:271)
... 7 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:400)
at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:150)
at org.elasticsearch.hadoop.hive.HiveFieldExtractor.extractField(HiveFieldExtractor.java:44)
at org.elasticsearch.hadoop.serialization.field.ConstantFieldExtractor.field(ConstantFieldExtractor.java:32)
at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:73)
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:77)
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:53)
at org.elasticsearch.hadoop.hive.EsSerDe.serialize(EsSerDe.java:128)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:262)
... 7 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
@costin costin closed this as completed in 57183bb Oct 3, 2014
@costin
Copy link
Member

costin commented Oct 3, 2014

Hi,

The exception is caused by the ttl field which indicates a value 5m and not an actual field. Thus it should be <5m>. The code has been improved to validate the input and avoid the NPE.

Thanks,

costin added a commit that referenced this issue Oct 3, 2014
@costin
[HIVE] Validate field extraction
e479bc2 
Double check JSON input (which has a different schema)

Relates #259
costin added a commit that referenced this issue Oct 3, 2014
@costin
[HIVE] Validate field extraction
21b9841 
Check whether the field indicated by the user is present in the table
schema to avoid NPE.

Fix #259

(cherry picked from commit 57183bb)
costin added a commit that referenced this issue Oct 3, 2014
@costin
[HIVE] Validate field extraction
36e4621 
Double check JSON input (which has a different schema)

Relates #259

(cherry picked from commit e479bc2)
costin added a commit that referenced this issue Oct 3, 2014
@costin
[HIVE] Validate field extraction
6c341d6 
Eliminate table clash in tests
Relates #259
costin added a commit that referenced this issue Oct 3, 2014
@costin
[HIVE] Validate field extraction
85702a9 
Eliminate table clash in tests
Relates #259

(cherry picked from commit 6c341d6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug enhancement :Hive v2.0.2 v2.1.0.Beta2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @snarla