Running unit tests for UnpackedAvroSource result

[mariussoutier]

This pull requests reproduces the issue I'm experiencing with testing AvroUnpackedSource via JobTest.

In sbt, you can run the test via test-only com.twitter.scalding.avro.AvroUnpackedSourceTest, resulting in a java.lang.IndexOutOfBoundsException: Index: 12, Size: 1.

I included the generated Avro classes to avoid having to integrate Avro into the sbt build.

[johnynek]

@ccsevers Can you take a look at this? Would be great to get this branch green.

[ccsevers]

Will have a look in the next day or two. Still a touch jet lagged.

[ccsevers]

Just a quick update, I haven't forgotten this but have been at essentially a conference all week after returning from overseas and haven't had the time to work this out. I'll get to it as soon as I can.

[ccsevers]

So I think what is going on here is that the Memory tap that gets used for testing is bypassing the unpacking step that happens in the cascading.avro scheme. If I add another test for the PackedAvroScheme it passes fine. I believe I've seen this same behavior with SequenceFiles (unpacking to (k,v) tuple)
@johnynek Have you run into this before?

I'll modify the tests to read/write as full jobs (rather than job test) and see that the behavior is correct.

[johnynek]

Yes, it is hard to test sources with the JobTest framework as it is actually mocking them. Maybe that is the whole issue here. I'm not sure of an easy fix. The memory source tap just serializes the data into the jobconf, and in this case, it will serialize the avro record. So, it won't unpack it on the other side in the test, so there will just be a single record, not a tuple. We should make this more testable, but it will require a bit of design work on how to get the same unpacking at test time.

Also note here is an error:

[error] x Running M/R jobs on Avro data should
[error]   x work for unpacked sources
[error]     Exception raised on property evaluation.
[error] > ARG_0: {"id": 3886654277753488978, "otherId": "", "date": "", "result": 1, "a": "", "b": "", "c": "", "d": "", "e": "", "f": "", "g": -432254720, "h": "", "price": -1.2344350315579075E307, "j": -8.644748785000766E307, "k": "", "l": -8.774343764295682E307, "m": -1917553931, "n": -1, "o": 582428282, "p": "", "q": -1201171014, "r": -1095342064, "s": -42409834, "t": -2147483648, "u": 0, "v": 2147483647, "w": 1.0, "x": -4.245805910546988E306, "y": 0.0, "z": 1.0, "aa": -7.194589934233269E307, "bb": 8.988465674311579E307, "cc": -1.464803824394378E307, "dd": -192812025, "ee": 1, "ff": -1, "gg": 2147483647, "hh": 1, "ii": -2118042387}
[error] > Exception: java.lang.ClassNotFoundException: com.twitter.scalding.avro.SimpleUnpackedAvroJob (AvroUnpackedSourceTest.scala:48)
[error] Failed: Total 2, Failed 2, Errors 0, Passed 0
[error] Failed tests:
[error]     com.twitter.scalding.avro.AvroUnpackedSourceTest

[ccsevers]

Do you think a reasonable workaround for now is to skip JobTest and just run the job and read the output?

[ccsevers]

Okay I can make this green but it's a bit ugly:
https://gist.github.com/ccsevers/9538254

[mariussoutier]

Thanks, I will try to use that for the time being. A cleaner unit testing solution would be great however.

[jcoveney]

To test integration stuff like this look at the PlatformTest class and use
of the MiniMRClustrr from hadoop

El lunes, 21 de julio de 2014, Marius Soutier notifications@github.com
escribió:

Closed #790 #790.

—
Reply to this email directly or view it on GitHub
#790 (comment).