Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import to pig fails when JSON contains an array #36

Closed
bwmeier opened this issue Apr 16, 2013 · 10 comments
Closed

Import to pig fails when JSON contains an array #36

bwmeier opened this issue Apr 16, 2013 · 10 comments
Labels
v1.3.0.M1

Comments

@bwmeier
Copy link

bwmeier commented Apr 16, 2013

When the JSON in the result set contains an array, then Pig 10.0 fails during internal serialization with an exception trace similar to the following. It appears that the array is being serialized as a Byte[] at some level, and Pig cannot handle that.

java.lang.RuntimeException: Unexpected data type [B found in stream. Note only standard Pig type is supported when you output from UDF/LoadFunc
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:559)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
    at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.impl.io.InterRecordWriter.write(InterRecordWriter.java:73)
    at org.apache.pig.impl.io.InterStorage.putNext(InterStorage.java:87)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:264)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
@costin
Copy link
Member

costin commented Apr 18, 2013

Can you upload a small snippet of your data set so I can replicate the problem on my side? Just upload it somewhere since I'm not sure Github can handle attachments and posting here might cause whitespace problems.

@bwmeier
Copy link
Author

bwmeier commented Apr 18, 2013

I added a short (1 result) json document that shows the result from an elasticsearch query on my system. The result set contains an array in lines 19, 22, and 23, any of which will cause the problem.

https://github.com/bwmeier/elasticsearch-hadoop/blob/master/elasticsearch_array_result.json

If you have any questions about the result, let me know.

@tzolov tzolov mentioned this issue Apr 19, 2013
Closed
@costin costin closed this as completed in 48358d6 Apr 19, 2013
@costin
Copy link
Member

costin commented Apr 19, 2013

@bwmeier the problem should be fixed in master. As @tzolov pointed out, there was a bug in parsing arrays into proper Writables causing the default, byte array serialization, to occur.
This should be now fix. Please try it out and let me know if that's not the case.

@bwmeier
Copy link
Author

bwmeier commented Apr 19, 2013

The error's changed slightly, but not significantly. I now get the following exception trace, which again looks like the array processing, just with the ArrayList type instead of the Byte[]. I suspect that pig needs a translation from ArrayList to DataBag, or something of the sort, since the message is related to "standard Pig type".

2013-04-19 12:12:04,057 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Unexpected data type java.util.ArrayList found in stream. Note only standard Pig type is supported when you output from UDF/LoadFunc
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:559)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
    at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.impl.io.InterRecordWriter.write(InterRecordWriter.java:73)
    at org.apache.pig.impl.io.InterStorage.putNext(InterStorage.java:87)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)

@costin
Copy link
Member

costin commented Apr 19, 2013

Understood - I've reopened the issue and will try to address it by next week.

@costin costin reopened this Apr 19, 2013
@costin costin closed this as completed in 8e49d4f May 3, 2013
@bwmeier
Copy link
Author

bwmeier commented May 3, 2013

Thanks @costin, I appreciate the work. I'll test this out when I get the chance, but it won't be for a few days :-)

@costin
Copy link
Member

costin commented May 4, 2013

Hi Boyd,

I think I've found (and fixed) the issue. Pig complex types were not handled properly when reading them back from Pig. Note that the conversion (not just for Pig but Hive, M/R etc...) will be overhauled however it should be working just fine now - the only issue is that bags are converted to tuples when are deserialized (and that's because we don't hold yet any extra information to allow us to differentiate between tuple and bags).
However this is something we plan to fix in the future.

It would be great if you could try out the latest master - see the readme on how to get the latest nightly build (I've just pushed one right now: http://build.elasticsearch.org/browse/ESHADOOP-NIGHTLY-40

@bwmeier
Copy link
Author

bwmeier commented May 14, 2013

Hi Costin, I've been on vacation since the 3rd, so I'm just now getting the
chance to look at this. I'll test it out and get back to you :-)

On Sat, May 4, 2013 at 3:19 AM, Costin Leau notifications@github.comwrote:

Hi Boyd,

I think I've found (and fixed) the issue. Pig complex types were not
handled properly when reading them back from Pig. Note that the conversion
(not just for Pig but Hive, M/R etc...) will be overhauled however it
should be working just fine now - the only issue is that bags are converted
to tuples when are deserialized (and that's because we don't hold yet any
extra information to allow us to differentiate between tuple and bags).
However this is something we plan to fix in the future.

It would be great if you could try out the latest master - see the readme
on how to get the latest nightly build (I've just pushed one right now:
http://build.elasticsearch.org/browse/ESHADOOP-NIGHTLY-40

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-17429716
.

@bwmeier
Copy link
Author

bwmeier commented May 20, 2013

Costin, the fix works. However, the performance is pretty poor - I was able to process 37500 records and write them to JsonStorage, but it took about 45 minutes, I'm not sure if it's related to the fix or not.

I tested the version of the fix pointed to by that commit, I have not tested anything later than that.

@costin
Copy link
Member

costin commented May 20, 2013

Note that there's an upcoming serialization improvement for writing to ES coming hopefully in the next few days.
It sounds though as this is not what you're doing - can you share your test (through a gist/email/pm maybe)?
You mention JsonStorage but I'm not sure why you would want to use since it adds the overhead of converting to JSON which we already handle.
Can you explain your use case more?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
v1.3.0.M1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @bwmeier