You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
users = LOAD 'hbase://TABLE'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'info:* crm:* synonymes:*', '-loadKey=true')
AS (id:chararray, info:map[], crm:map[], synonymes:map[]);
STORE usersINTO 'test/test_users' USING ESStorage();
In this situtation it may happen that some of my maps are empty because my data matrix is sparse. Say info and synonymes are ok and crm is empty some times.
Pig will throw this error:
java.lang.NullPointerException
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:103)
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:128)
looking at the line
case DataType.MAP:
ResourceSchema nestedSchema = field.getSchema();
ResourceFieldSchema[] nestedFields = nestedSchema.getFields();
I'm guessing this happens because my map is empty as as such there are no schema and fields.
regards
The text was updated successfully, but these errors were encountered:
A quick look around shows this could happen with tuples and bags as well.
A quick compile and tested it to look for fields only if the nestedSchema is not null.
If it is null it will not enter the loop so the generator will write an empty object.
Not the nicest hack but it works in my case.
Thanks for the report - I've just pushed a fix for all cases (along with an integration test). It's unfortunate that Pig doesn't preserve the structure even for empty objects but the writer can cope with this case now.
Cheers!
Hello everyone
I have come accross this error in my pig to elsaticsearch import where having an empty map will throw a null pointer exception.
to be more precise I have a simple pigscript:
DEFINE ESStorage org.elasticsearch.hadoop.pig.ESStorage('es.host=thehost');
users = LOAD 'hbase://TABLE'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'info:* crm:* synonymes:*', '-loadKey=true')
AS (id:chararray, info:map[], crm:map[], synonymes:map[]);
STORE usersINTO 'test/test_users' USING ESStorage();
In this situtation it may happen that some of my maps are empty because my data matrix is sparse. Say info and synonymes are ok and crm is empty some times.
Pig will throw this error:
java.lang.NullPointerException
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:103)
at org.elasticsearch.hadoop.pig.PigValueWriter.write(PigValueWriter.java:128)
looking at the line
case DataType.MAP:
ResourceSchema nestedSchema = field.getSchema();
ResourceFieldSchema[] nestedFields = nestedSchema.getFields();
I'm guessing this happens because my map is empty as as such there are no schema and fields.
regards
The text was updated successfully, but these errors were encountered: