TAJO-906: Runtime code generation for evaluating expression trees.

[hyunsik]

This is still an ongoing work. I share it for reviewing an overall approach.

Note that this patch includes some third party library because the version of ow2.asm is conflict to Hadoop's one. So, we don't need to review ow2.asm.

Also, this patch still does not pass one unit test related to IntervalDatum. I'll fix it soon.

[hyunsik]

I've rebased and fixed many bugs. Even though there are some trivial issues to do, this patch is ready to be reviewed.

[hyunsik]

I've updated the patch. I've improved TajoTestingCluster to take system properties as follows:

mvn clean install -DCODEGEN=true

If CODEGEN is a session variable, it will be applied to QueryContext instance used in all unit tests. So, in order to test the code generation feature, you should give -DCODEGEN=true when you execute mvn install. It can be used for other session variables too.

For test for real queries, you need to set a session variable CODEGEN as follows:

tajo> \set CODEGEN true
tajo> 
tajo> SELECT .....

[blrunner]

Hi @hyunsik

This is a really great work. But I recommend for you to consider license.
If you add ASM project license to tajo license file and tajo notice file, it would not be a problem.

Cheers

[hyunsik]

Thanks @blrunner for the review. I've updated the follows according to ASF license policy.

• ASM is BSD license. So, according to Modifications to NOTICE, I didn't modify NOTICE file.
• Instead, I've added ASM license to only LICENSE file.
• Removed ASF standard license header from ASM source codes.
  • Please refer to Treatment of Third-Party Works

[blrunner]

I checked on ASF license deletion from ASM source codes. And I double-checked tajo LICENSE file.
'mvn clean install -DCODEGEN=true' finished successfully. And then I found that CODEGEN variable on tsql as follows:
{code:xml}
default> \set CODEGEN true
default> \set
'CURRENT_DATABASE'='default'
'CODEGEN'='true'
'SESSION_ID'='f902f118-2649-46df-97ec-f15c2f0c75f5'
'USERNAME'='blrunner'
'SESSION_LAST_ACCESS_TIME'='1408801963706'
{code}

But I have a question about this patch. Is there any way of checking runtime code generation on local cluster?

[hyunsik]

Hi @blrunner,

If CODEGEN session variable is enabled, all queries will work by using code generation. But, currently, we cannot expect its performance improvement. The code generation is designed to avoid Datum objects creations during any computations and reduce interpretation overheads like branches.

But, in order to keep the compatibility against our current Tuple and Datum mechanism, code generation feature still create lots of Datum objects; currently it creates objects two times for compatibility. Currently, I'm working new tuple structure using direct memory, which uses a sequence of bytes as a row blocks instead of an array of Datum objects. After than, it will give big performance benefits.

Thanks!

[blrunner]

+1

Thank you for your detailed comments.
I'm also looking forward for it to improve tajo performance. :)