Finish support for Pig 0.12 #69
Conversation
PIG-3367 has introduced a new keyword "ASSERT" in version 0.12.0 and thus the same named UDF can not be used any more. I've renamed the UDF to "AssertUDF" to enable the UDF in pig 0.12.0 and provided a compatible layer to not break users that are still using Pig 0.11.0.
PIG-3269 has introduced a new keyword "IN" in version 0.12.0 and thus the same named UDF can not be used any more. I've renamed the UDF to "InUDF" to enable the UDF in pig 0.12.0 and provided a compatible layer to not break users that are still using Pig 0.11.0.
|
What are the issues you observe with Assert and In? When I tried running the unit tests earlier with Pig 0.12.0 I thought all the unit tests passed. |
|
Hi Matt, ant clean test -Dtest.methods=datafu.test.pig.util.AssertTests.shouldNotAssertWithoutMessageOnOne -Dpig.version=0.12.0 It will fail with following exception: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. <line 1, column 7> mismatched input 'ASSERT' expecting IDENTIFIER This make complete sense as we are having definition using "assert" which is a keyword now, so I've tried to change the test define ASR instead, but that did not quite help as now Pig seems to be unhappy with the "Assert" part of the UDF definition (datafu.pig.util.Assert): org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. <line 1, column 28> Syntax error, unexpected symbol at or near 'Assert' I've tried couple of experiments, such as removing the "define" and using the Assert UDF directly in the FILTER operator but nothing has worked for me. If you have any other idea, please don't hesitate and let me know. I'll be more than happy to refactore proposed changes. |
|
Okay yea I just tried this and got the same error. I think your proposal makes sense. Let me think on it for a bit. |
|
Please take your time sir! |
|
Thank you Matt! |
Pig 0.12.0 has added several new keywords such as "Assert" or "In" (via PIG-3367 and PIG-3269). This is causing issues to DataFu UDFs with the same name. I wasn't able to find any other solution other then renaming the UDFs in question. I do understand that this is very intrusive change, so I've provided deprecated ones for backward compatibility - e.g users on Pig 0.11.0 will still be able to use the Assert or In as they are now.
After applying this final set of changes I was able to run all unit tests against Pig 0.12.0 without any issues!