Getting error in EB LzoProtobufB64LinePigStorage storefunc.

[viswaj]

Hi Raghu,

Facing error in EB storefunc,

3779 at com.test.sample.data.Sample$Testing.getSerializedSize(Sample.java:19967)
3780 at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
3781 at com.twitter.elephantbird.mapreduce.io.ProtobufConverter.toBytes(ProtobufConverter.java:73)
3782 at com.twitter.elephantbird.mapreduce.io.ProtobufConverter.toBytes(ProtobufConverter.java:15)
3783 at com.twitter.elephantbird.mapreduce.output.LzoBinaryB64LineRecordWriter.write(LzoBinaryB64LineRecordWriter.java:42)
3784 at com.twitter.elephantbird.mapreduce.output.LzoBinaryB64LineRecordWriter.write(LzoBinaryB64LineRecordWriter.java:26)
3785 at com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage.putNext(LzoProtobufB64LinePigStorage.java:55)
3786 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
3787 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
3788 at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:576)
3789 at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
3790 at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)

I have proto with multiple message blocks. How to access them in latest EB in storefunc of LzoProtobuffB64LinePigStore? I'm refer as follows,

com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.sample.data.Sample.Testing')

Earlier I used com.twitter.elephantbird.pig.proto.LzoProtobuffB64LinePigStore it works well(i.e., com.twitter.elephantbird.pig.proto.LzoProtobuffB64LinePigStore('Testing')), but in latest version this class not found, even I loaded the below jars,

elephant-bird-hadoop-compat-4.4.jar
elephant-bird-pig-4.4.jar
elephant-bird-core-4.4.jar

Pls help.

[viswaj]

com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.sample.data.Sample$ Testing') have tried to access objects like this but same issue.

[viswaj]

Hi Raghu,

I'm facing this issue during store the result with the above format in pig.

[viswaj]

Please help on this.

[rangadi]

This looks like protobuf version mismatch. What is the is the version your
protobuf is generated with?

I think with Hadoop you might end up with ver 2.4.1 on the mappers and
reducers.

On Sunday, March 9, 2014, Viswanathan Jayamoorthy notifications@github.com
wrote:

Please help on this.

Reply to this email directly or view it on GitHubhttps://github.com/kevinweil/elephant-bird/issues/377#issuecomment-37148724
.

composed on mobile

[viswaj]

Hi Raghu,

I'm using protobuf-2.4.1 version for generating the data. Now I'm using the hadoop2.x and it has protobuf version of 2.5.0 is that not backward compatible? Or do I need to copy the protobuf-2.4.1 jar in hadoop lib?

[viswaj]

Hi Raghu,
Please help.

[viswaj]

Hi Raghu,

It looks I'm unable to use this class com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage in pig store method.

[sritchie]

@viswaj, just a note - @raghu responded last night before he want to bed, and hasn't responded in the last twelve hours because he's asleep :) It's currently 8:43 AM in California. Give him some time, I'm sure he'll respond soon.

[sritchie]

And 2.5.0 is almost certainly not backward compatible. That bump from 2.4 -> 2.5 signals incompatibility.

[viswaj]

yeah, I knew that he will respond always. Just posted the errors to let him know.

[sritchie]

I guarantee he's going to ask - "Why are you unable to use the class?" What error are you seeing? Can you provide a stack trace, or more detailed information to help diagnose?

[rangadi]

Yeah, these protobuf versions are not compatible.

@viswaj can you compile your proto classes with 2.5? ElephantBird should work fine with either versions.

[ Just in case you want to use LzoProtobufBlockPigStorage instead of LzoProtobufB64LinePigStorage, you do need latest changes in elephant-bird master, it was recently updated to work with protobuf 2.5]

[viswaj]

Hi Raghu,

Thanks for the info. Have fixed the compatible issue and it works well. I able to read the data using com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage by parsing the proto msg class.

The problem is I could not able to use this class(com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage) for storing the result data with pig store function. I need the data to be store as encoded with lzo format.

[rangadi]

You listed the same class for twice in your message. LzoProtobufB64LinePigStorage is only for storing. For reading, you would use LzoProtobufB64LinePigLoader (ends with Loader).

If you don't have protobuf version mismatch, both loader and storage should work. As Sam suggested, please post your new error, and if you can, post your pig script as well.

[viswaj]

Hi Raghu,

I'm using the loader class only for reading and it works fine.
For storing i'm using storage class only and store example as below.

eg.,

store d into '/tmp/sampledata' using com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.log.testing.Store$TestStore');

Hope I need to pass the complete proto class in storage arg.

[isnotinvain]

long shot but.... I think you need to escape the $ in your pig script as $
Otherwise it's going to look for a variable called $TestStore right?
Are you getting:
Encountered IOException.
org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined
parameter : TestStore
?

On Mon, Mar 10, 2014 at 10:19 AM, Viswanathan Jayamoorthy <
notifications@github.com> wrote:

Hi Raghu,

I'm using the loader class only for reading and it works fine.
For storing i'm using storage class only and store example as below.

eg.,

store d into '/tmp/sampledata' using
com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.log.testing.Store$TestStore');

Hope I need to pass the complete proto class in storage arg.

Reply to this email directly or view it on GitHubhttps://github.com/kevinweil/elephant-bird/issues/377#issuecomment-37208461
.

[rangadi]

store d into '/tmp/sampledata' using com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.log.testing.Store$TestStore');

What is the error you see with the above store?

[viswaj]

Thanks guys.

store d into '/tmp/sampledata' using com.twitter.elephantbird.pig.store.LzoProtobufB64LinePigStorage('com.test.log.testing.Store.TestStore');

It's resolved by changing as above and able to store the data in message proto.