Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multifield Mapping wihout default field causes a Nullpointerexception #132

Closed
nahap opened this issue Feb 5, 2014 · 5 comments
Closed

Multifield Mapping wihout default field causes a Nullpointerexception #132

nahap opened this issue Feb 5, 2014 · 5 comments
Labels
bug :Rest v1.3.0.M2

Comments

@nahap
Copy link

nahap commented Feb 5, 2014

When you have a Multifield Mapping that does not have a default then selecting from hive will result in a Nullpointerexception. Here is how to reproduct this. (I tested this on Elasticsearch 0.90.10 with the current master of elasticsearch-hadoop.

curl -XDELETE http://localhost:9200/test_index/product/_mapping
curl -XDELETE http://localhost:9200/test_index

curl -XPUT http://localhost:9200/test_index
curl -XPUT http://localhost:9200/test_index/product/_mapping -d '
{
    "product": {
        "properties": {
            "mf_test" : {
                "type" : "multi_field",
                "fields" : {
                    "name" : {"type" : "string", "index" : "analyzed"},
                    "untouched" : {"type" : "string", "index" : "not_analyzed"}
                }
            }
        }
    }
}'

 curl -XPUT 'http://localhost:9200/test_index/product/1' -d '
 {
    "id": 1,
    "mf_test" : "anything"
 }'

CREATE EXTERNAL TABLE test.products(
    product_id BIGINT,
    attribute STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'test_index/product/_search?q=*',
              'es.nodes'='localhost',
              'es.port'='9200',
              'es.nodes.discovery' = 'false',
              'es.mapping.names' = 'product_id:id,attribute:mf_test');

select * from test.products;

you get the exception:

14/02/05 11:38:35 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:551)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:489)
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1472)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.NullPointerException
    at org.elasticsearch.hadoop.rest.dto.mapping.Field.parseField(Field.java:118)
    at org.elasticsearch.hadoop.rest.dto.mapping.Field.parseField(Field.java:133)
    at org.elasticsearch.hadoop.rest.dto.mapping.Field.parseField(Field.java:133)
    at org.elasticsearch.hadoop.rest.dto.mapping.Field.parseField(Field.java:65)
    at org.elasticsearch.hadoop.rest.RestRepository.getMapping(RestRepository.java:224)
    at org.elasticsearch.hadoop.mr.EsInputFormat.getSplits(EsInputFormat.java:391)
    at org.elasticsearch.hadoop.hive.EsHiveInputFormat.getSplits(EsHiveInputFormat.java:93)
    at org.elasticsearch.hadoop.hive.EsHiveInputFormat.getSplits(EsHiveInputFormat.java:44)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:381)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:515)
    ... 14 more

If you put a default field in the mapping, then there is no error.

curl -XDELETE http://localhost:9200/test_index/product/_mapping
curl -XDELETE http://localhost:9200/test_index
curl -XPUT http://localhost:9200/test_index
curl -XPUT http://localhost:9200/test_index/product/_mapping -d '
{
    "product": {
        "properties": {
            "mf_test" : {
                "type" : "multi_field",
                "fields" : {
                    "name" : {"type" : "string", "index" : "analyzed"},
                    "untouched" : {"type" : "string", "index" : "not_analyzed"},
                    "mf_test" : {"type" : "string", "index" : "analyzed"}
                }
            }
        }
    }
}'
 curl -XPUT 'http://localhost:9200/test_index/product/1' -d '
 {
    "id": 1,
    "mf_test" : "anything"
 }'

select * from test.products;

works as expected when the default field is set.
IMPORTANT: make sure you test this with 0.90.10, because 1.0.0.RC1 has the following issue with multifield mappings : elastic/elasticsearch#5016
@costin
Copy link
Member

costin commented Feb 5, 2014

I've pushed a fix to master which should fix your issue.
Can you please try it out and let me know whether it works for you or not. Basically in case of a non-default field, we'll look at the types of the subfields - if they are all the same type, we use that. if not, we throw an exception (since we don't know what type to use - it's an ambiguous definition).

Cheers!

@nahap
Copy link
Author

nahap commented Feb 5, 2014

This works fine now, no exception and we get the proper data returned. Thanks!

@costin
Copy link
Member

costin commented Feb 5, 2014

Great - thanks again for the report and feedback!

@costin costin closed this as completed Feb 5, 2014
@costin costin added v1.3.0.M2 and removed mr labels Feb 6, 2014
@pgurani123
Copy link

hi Costin, i am getting a different error while trying to fetch data through hive -
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.rest.dto.mapping.Field
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:76)

Any idea what could be the issue here?

@costin
Copy link
Member

costin commented Aug 28, 2015

@pgurani123 Instead of commenting on a bug closed a year ago, create a new one or better yet, using the dedicated forum?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug :Rest v1.3.0.M2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nahap @costin @pgurani123