TAJO-1344: Python UDF support

[jihoonson]

I found some problems of version confliction when using Jython.
So, I used another approach of using pipe.
Most codes are borrowed from Pig.

[jihoonson]

Ok. I think that this patch is ready for review.

To address @hyunsik's comment, I added a class called FunctionInvoke. This class describes how the functions are executed.

On executing Python scripts, I used the approach of using an external UDF controller that is responsible for executing python scripts as commented above. When a submitted query involves one or more python UDFs, several UDF controllers are executed to compute UDFs. Input/output tuples are transmitted via stdio. This approach may have an issue on performance, but I think it is inevitable without using Jython.

Currently, the controller is executed for each Python functions. That is, if a query involves 5 Python functions even some of them are same, at least 5 different controllers are executed during query processing. I chose this architecture due to its simplicity.

Here are some highlights of changes.

• AnyDatum is used to support Python's dynamic typing.
• PythonScriptEngine is responsible for maintaining the external controller process. To reduce overhead, the controller should be forked only when UDFs are actually evaluated. In this patch, there are three points where the controller is forked.
  • Constant folding optimization in Tajo master: During constant folding, some UDFs can be evaluated. If necessary, controllers are forked and immediately destroyed after evaluation.
  • Non-from query execution in Tajo master. If the query involves Python UDFs, controllers are forked during query processing.
  • Task execution in worker: If the plan of a stage involves Python UDFs, controllers are forked (destroyed) when a task starts up (shuts down). Due to the simplicity, I chose this architecture rather than sharing controllers among multiple tasks via ExecutionBlockSharedResource.
• Refactoring the EvalNode::bind() function. This function now receives EvalContext in addition to Schema. EvalContext can contain some information given at runtime such as ScriptEngine started by each task.

For reviewers, I apologize for a large patch. But many changes are related to just refactoring of the bind() function and renaming some functions.
Thanks.

[jinossy]

Great!! @jihoonson
I will start review. If you fix the findbug warnings in your codes, It would be better

[hyunsik]

Thank you all guys for your efforts. I also check only some design points.

[jihoonson]

Thanks @jinossy and @hyunsik. I've fixed some findbugs warnings related to my patches.

[hyunsik]

In my opinion, 'user-defined function' would be a better word than 'optional function'

[hyunsik]

I leaved some trivial comments. The patch looks good to me. It's a great job.

Here is my additional comments. You try to change the signature EvalNode::bind to take EvalContext instance in order to get an instance of a launched script engine.

Even though this refactoring is already a breaking change. EvalNode still needs more information about task. In this chance, it would be great if we refactor EvalNode to take more meta context including TajoConf, task attempt information, and shared resources of workers. Then, we can probably remove OverridableConf parameter from all constructors of EvalNode.

If you are concerned with a large patch, we can do this work in another jira. But, it will cause the second breaking change. It is also not good as much as a large patch. You can choose either two breaking changes or a large patch. It's up to you.

[jihoonson]

Thanks @hyunsik. I've changed the function name and removed QueryContextUtil according to your comment.

Regarding on refactoring bind() function, I think that it would be better to work in another jira. For this issue, we first should decide what information are required to be passed to EvalNode. I booked another jira (https://issues.apache.org/jira/browse/TAJO-1566).

[hyunsik]

I got your point and your plan. Please keep going. Here is my +1.

[jihoonson]

I've tested on my laptop.
@jinossy, do you have any other opinions?

[jinossy]

No, Looks good to me. Here is my +1
Thank you for your work