ClassCastException VIntWritable and java.lang.Integer #57
Comments
|
What's your Hive/Hadoop environment (versions, OS, etc...)? |
|
Now I am using Mac OS, with Hadoop 1.2.0,Hive 0.11 and ES 0.90.2. I tried with Hadoop 0.23 as well, but I caught the same problem. I tried in linux too with a cluster and the same error appeared. |
|
The ESSerDE just have a parser implementation to STRUCT and ARRAY. When values like String come from ES, they type is Text. So when Hive convert data Text to String, probably call toString method. But with Long or Integer values, Hive tries to cast to a specific class (in this case Integer). I made some changes in parser ESSerDe and they work fine. |
|
Marcelo, ESSerDe is currently uncomplete and it will be addressed by the mapping feature (wip). I've tried to fix it but in the current format (Writable to Hive) it cannot be done in a reliable way. |
I have a type in ES and in Hive I added a EXTERNAL TABLE pointing to ES. Something like this:
CREATE EXTERNAL TABLE user (userId INT, userRoles ARRAY<STRUCT<roleId:INT, name:STRING>>) STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler' TBLPROPERTIES('es.resource'='samples/user/_search?q=*');
My mapping in ES sounds like this:
{
"user" : {
"properties" : {
"userId" : {
"type" : "int"
},
"userRoles" : {
"properties" : {
"roleId" : {
"type" : "int"
},
"name" : {
"type" : "string"
}
}
}
}
}
}
So when a execute a SELECT * FROM user I caught:
2013-07-08 09:09:17,648 ERROR CliDriver (SessionState.java:printError(386)) - Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.io.VIntWritable cannot be cast to java.lang.Integer
java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.io.VIntWritable cannot be cast to java.lang.Integer
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.VIntWritable cannot be cast to java.lang.Integer
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.get(JavaIntObjectInspector.java:39)
at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:201)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:428)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:382)
at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:71)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:366)
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:490)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
... 11 more
When I only use strings, works fine. I tried the master branch and the maven repositories available releases, but none works.
The text was updated successfully, but these errors were encountered: