Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Field Mapping exception #231

Closed
b0c1 opened this Issue Jul 14, 2014 · 7 comments

Comments

Projects
None yet
2 participants
@b0c1
Copy link

commented Jul 14, 2014

Hi guys,
I have a little problem when I want to read ES.
My application receive json from external source, enrich and merge the enriched value with the raw data. I got a problem when I want to read back the data for aggregation,

The problem is in:
org.elasticsearch.hadoop.serialization.dto.mapping.Field.add(Field.java:110)

Here the field.properties() is null so I got null pointer exception.

How can I solve this (or how can I get the raw json from ES?)

@costin

This comment has been minimized.

Copy link
Member

commented Jul 15, 2014

It looks like either the JSON is invalid or the parsing fails for some reason. Do you have a sample JSON that causes the issue? Try turning on logging to see the data that goes through the wire.

costin added a commit that referenced this issue Jul 15, 2014

Eliminate NPE
relates to #231
@costin

This comment has been minimized.

Copy link
Member

commented Jul 15, 2014

Hi,

I've pushed a potential fix that avoid the NPE (which should not occur in the first place) so please try out the 2.x branch or the upcoming 2.0.1.BUILD-SNAPSHOT and report back. Even if it works, please send a doc that created the NPE in the first place - I'd like to take a closer look at it and make sure it is properly parsed.

@costin costin added bug labels Jul 15, 2014

@b0c1

This comment has been minimized.

Copy link
Author

commented Jul 16, 2014

Now I not get any NPE, meanwhile I have a problem. In elastic search I got a very large dynamic data (so I can't create type mapping). Can I read the elasticsearch result as raw json? (the ES mapping type is wrong sometime, example It's got "" where it's need a number, try to put the double value to long etc)

@costin

This comment has been minimized.

Copy link
Member

commented Jul 18, 2014

When dealing with mapping issues, it's best to use templates where possible. Even in dynamic cases, it allows you to lock in some fields and eliminate some misconfiguration. You can read a document (depending on the integration) in raw format by mapping it to a Text/String/chararray.
Try however disabling es.field.read.empty.as.null first to see whether it fixes your issue.

@b0c1

This comment has been minimized.

Copy link
Author

commented Jul 18, 2014

Yeah but I have a lot field (200+) and it's dinamic (maybe come, or not, sometimes the sender add a new field) I do not want to store the incoming data as Text (maybe sometimes I want to filter to some field) but I want to read back the whole document as text in Hadoop. Now ES-Hadoop only support mapped data (read the es data into linked map). I want to simply read the whole record into ... simply Text (where I can get the required fields/etc with mapreduce using JSON4s forexample...

@costin

This comment has been minimized.

Copy link
Member

commented Jul 18, 2014

Understood. you are looking for the equivalent of es.input.json but for output. That is possible but it is not streamlined.
I'll raise a separate issue for this and close the current one.

@b0c1

This comment has been minimized.

Copy link
Author

commented Jul 18, 2014

Thanks a lot

costin added a commit that referenced this issue Jul 31, 2014

Eliminate NPE
relates to #231

(cherry picked from commit 558f1fa)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.