Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pig empty Maps #89

Closed
nmaillard opened this issue Oct 2, 2013 · 3 comments
Closed

Pig empty Maps #89

nmaillard opened this issue Oct 2, 2013 · 3 comments

Comments

@nmaillard
Copy link

Hello everyone

I have come accross this error in my pig to elsaticsearch import where having an empty map will throw a null pointer exception.

to be more precise I have a simple pigscript:

DEFINE ESStorage org.elasticsearch.hadoop.pig.ESStorage('es.host=thehost');

users = LOAD 'hbase://TABLE'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'info:* crm:* synonymes:*', '-loadKey=true')
AS (id:chararray, info:map[], crm:map[], synonymes:map[]);

STORE usersINTO 'test/test_users' USING ESStorage();

In this situtation it may happen that some of my maps are empty because my data matrix is sparse. Say info and synonymes are ok and crm is empty some times.

Pig will throw this error:

java.lang.NullPointerException
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:103)
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:128)

looking at the line

case DataType.MAP:
ResourceSchema nestedSchema = field.getSchema();
ResourceFieldSchema[] nestedFields = nestedSchema.getFields();

I'm guessing this happens because my map is empty as as such there are no schema and fields.

regards

@nmaillard
Copy link
Author

A quick look around shows this could happen with tuples and bags as well.

A quick compile and tested it to look for fields only if the nestedSchema is not null.
If it is null it will not enter the loop so the generator will write an empty object.
Not the nicest hack but it works in my case.

@costin costin closed this as completed in 3b84f6d Oct 2, 2013
@costin
Copy link
Member

costin commented Oct 2, 2013

Thanks for the report - I've just pushed a fix for all cases (along with an integration test). It's unfortunate that Pig doesn't preserve the structure even for empty objects but the writer can cope with this case now.
Cheers!

@nmaillard
Copy link
Author

thanks a lot and cheers on the responsivness

@costin costin added bug labels Mar 6, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants