TAJO-1493: Make partition pruning based on catalog informations.

[blrunner]

Not yet implemented unit test cases.

[blrunner]

I implemented CatalogStore::getPartitionsByDirectSql which execute a select statement to CatalogStore for searching right partition directories. But current tajo testing cluster use MemStore for default CatalogStore and I couldn't execute the select statement because MemStore just retain all catalog informations to HashMap. If you want to verify this patch, you can test as following:

mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore

[blrunner]

I found that this patch run as expected with HiveCatalogStore and MySQLStore on my testing cluster. And simple query response had been reported as following:

• Table schema:

create table partitioned_lineitem (L_SUPPKEY bigint, L_LINENUMBER bigint,
L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text,
L_SHIPDATE text, L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text)
partition by column (L_ORDERKEY bigint, L_PARTKEY bigint)

• Partition numbers: 100,000
• Select statement: select * from partitioned_lineitem limit 10;
• Response time:
• previous rewriter: 15 ~ 16 sec
• improved rewriter: 12 ~ 13 sec

Honestly, I didn't implement unit test cases for executing queries because current almost tajo unit cases operate on MemStore. If we apply DerbyStore to some unit test cases for physical operator, we would make a lot of effort. It seems not to be the scope of this patch. So, I just added unit test cases for verifying direct sql. But if you want to test this patch with build commands, you can test with -Dtajo.catalog.store.class parameter as following:

mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.DerbyStore

[hyunsik]

You may forget the removal of commented out lines.

[hyunsik]

Catalog API is public. In terms of this point, this API design does not make sense due to the following problems:

• Users should know table schemas, their relationship, which underlying persistent storage.
  • Users should make SQL statements for all persistent storages.
• This approach is very prone to SQL injection.

The solution is that you can use some intermediate representation, using string or java objects. They should be transformed into SQL statements depending on persistent storages in each catalog driver.

[blrunner]

Hi @hyunsik

I've implemented partition API using algebra expressions. For this feature, I implemented some visitors as following:

• ScanQualConverter: This converts Quals of ScanNode to algebra expressions.
• PartitionFilterAlgebraVisitor: This build SQL statement for getting partition filters by visiting algebra expressions.
• PartitionFilterEvalNodeVisitor: This build SQL statement for getting partition filters by visiting EvalNodes.

[blrunner]

@hyunsik

Thank you for your detailed review and I've just reflected your comments.

[hyunsik]

If it is not used, you may do not need to add this method.

[blrunner]

Thanks @hyunsik

I reflected your comments and left some comments. :)

[hyunsik]

Could you rename TimeStamp to Timestamp?

[hyunsik]

Thank you for your work. The latest patch looks good to me. Here is my +1. I leaved one comment. You can commit it after reflecting my comment if you agree.

[blrunner]

@hyunsik

Thank you very much for your kind review. I've just reflected your comment. After finishing the travis CI build, I'll ship it. :)