TAJO-1952: Implement PartitionFileFragment #846

blrunner · 2015-11-05T08:59:11Z

This patch contains following modifications:

Remove partition paths from PartitionedTableScanNode
Implement PartitionedFileFragment
Separate a method for pruning partition paths from PartitionedTableRewriter to PartitionedTableUtil
Build a type by using partition name which contains partition keys and values

hyunsik · 2015-11-19T02:24:13Z

tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java

  /**
   * Generate the list of files and make them into FileSplits.
   *
   * @throws IOException
   */
-  public List<Fragment> getSplits(String tableName, TableMeta meta, Schema schema, Path... inputs)
+  public List<Fragment> getSplits(String tableName, TableMeta meta, Schema schema, String[] partitions, Path... inputs)


I'm going to give a suggestion. We need to keep each method for one purpose if possible. That approach would keep logic simpler.

hyunsik · 2015-11-19T02:42:57Z

Could you rebase it against the latest revision? I'd like to try the patch on my machine.

blrunner · 2015-11-19T02:57:42Z

@hyunsik

Thank you for your review. I've just rebased it against the latest version.

hyunsik · 2015-11-19T05:30:10Z

tajo-storage/tajo-storage-hdfs/src/main/proto/StorageFragmentProtos.proto

+  repeated string hosts = 5;
+  repeated int32 disk_ids = 6;
+  // Partition Name: country=KOREA/city=SEOUL
+  required string partitionName = 7;


In my opionion, partitionKeys would be more proper because the attribute includes concatenated partition keys.

blrunner · 2015-11-20T02:27:18Z

@hyunsik

Thank your for your review. I've just reflected your comments.

blrunner · 2015-11-23T09:34:39Z

I updated the patch as following:

Renamed PartitionedFileFragment to PartitionFileFragment
Recovered codes for pruning partition in PartitionedTableRewrtier
Recovered partition paths of PartitionedTableScanNode and added partition keys to PartitionedTableScanNode
Implemented PartitionContent which contains partition paths, partition keys, partition volume

For the reference, I've tried to maintain existing codes as far as possible.

hyunsik · 2015-11-24T02:31:48Z

tajo-plan/src/main/java/org/apache/tajo/plan/partition/PartitionContent.java

+  public PartitionContent() {
+  }
+
+  public PartitionContent(Path[] partitionPaths) {


unused constructor

blrunner · 2015-11-24T09:08:30Z

@hyunsik

Updated the patch as following:

Remove unused packages
Remove PartitionedTableScanNode of PlanProto
Remove unnecessary PartitionedTableScanNode usages of Repartitioner and PhysicalPlannerImpl

blrunner · 2015-11-25T07:22:26Z

Added unit test cases for PartitionedTableRewriter

hyunsik · 2015-12-01T05:48:18Z

tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileTablespace.java

+    }
+
+    List<Map.Entry<String, Integer>> entries = new ArrayList<>(hostsBlockMap.entrySet());
+    Collections.sort(entries, new Comparator<Map.Entry<String, Integer>>() {


It can be simplified with lambda.

blrunner · 2015-12-03T02:48:42Z

@hyunsik

Thank you for your detailed review. I reflected your comments.

blrunner · 2015-12-07T08:47:03Z

@hyunsik

I changed PartitionFileFragment by extending FileFragment because there are some codes for only FileFragment. For example, you can see cast codes as following:

  public FileScanner(Configuration conf, final Schema schema, final TableMeta meta, final Fragment fragment) {
    this.conf = conf;
    this.meta = meta;
    this.schema = schema;
    this.fragment = (FileFragment)fragment;
    this.tableStats = new TableStats();
    this.columnNum = this.schema.size();
  }

blrunner · 2015-12-10T06:16:37Z

Evaluated patch testing before and after PartitionFileFragment implementation as following:

Dataset : TPCH-100G
Tajo Cluster : 1 Master, 6 Workers
Queries: Q1, Q3, Q5, Q6, Q7, Q8, Q9, Q10
Tajo version for before PartitionFileFragment implementation
- 0.11.1-SNAPSHOT
Tajo version for after PartitionFileFragment implementation:
- 0.12.0-SNAPSHOT (partitions exist on catalog)
- 0.12.0-SNAPSHOT (partitions doesn't exist on catalog)

The results were same as following:

The number and order of execution blocks
The number of tasks in a each execution block
The number of rows in a result
All tuples in a result (excluded a few floating point value)

blrunner · 2016-01-05T09:16:54Z

The Travis CI Build seems like a fail by another reason. Here is the pre-commit report by lasted patch as follows. https://builds.apache.org/job/PreCommit-TAJO-Build/900//console

blrunner · 2016-01-17T06:23:01Z

@hyunsik

Is there a problem that needs to be fixed?

blrunner · 2016-03-21T16:28:50Z

@hyunsik

I updated this PR as following:

Remove the list of partition paths in PartitionedTableScanNode. It just contains the path of table root.
PartitionedTableRewriter just rewrite ScanNode to PartitionedTableScanNode in BaseLogicalPlanRewriteRuleProvider.
Actual partition pruning will be executed in Repartitioner by PartitionedTableRewriter.

hyunsik reviewed Nov 19, 2015
View reviewed changes

hyunsik reviewed Nov 24, 2015
View reviewed changes

blrunner changed the title ~~TAJO-1952: Implement PartitionedFileFragment~~ TAJO-1952: Implement PartitionFileFragment Nov 25, 2015

hyunsik reviewed Dec 1, 2015
View reviewed changes

blrunner mentioned this pull request Mar 18, 2016

TAJO-1891: Allow arbitrary partition path #984

Closed

blrunner mentioned this pull request Apr 4, 2016

TAJO-2111: Optimize Partition Table Split Computation for Amazon S3 #994

Closed

blrunner force-pushed the TAJO-1952 branch from e966984 to fd8f144 Compare May 13, 2016 06:47

blrunner closed this May 13, 2016

blrunner force-pushed the TAJO-1952 branch from ee14847 to 01d2fe2 Compare May 13, 2016 07:49

blrunner mentioned this pull request May 16, 2016

TAJO-1952: Implement PartitionFileFragment #1020

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TAJO-1952: Implement PartitionFileFragment #846

TAJO-1952: Implement PartitionFileFragment #846

blrunner commented Nov 5, 2015

hyunsik Nov 19, 2015

hyunsik commented Nov 19, 2015

blrunner commented Nov 19, 2015

hyunsik Nov 19, 2015

blrunner commented Nov 20, 2015

blrunner commented Nov 23, 2015

hyunsik Nov 24, 2015

blrunner commented Nov 24, 2015

blrunner commented Nov 25, 2015

hyunsik Dec 1, 2015

blrunner commented Dec 3, 2015

blrunner commented Dec 7, 2015

blrunner commented Dec 10, 2015

blrunner commented Jan 5, 2016

blrunner commented Jan 17, 2016

blrunner commented Mar 21, 2016

TAJO-1952: Implement PartitionFileFragment #846

TAJO-1952: Implement PartitionFileFragment #846

Conversation

blrunner commented Nov 5, 2015

hyunsik Nov 19, 2015

Choose a reason for hiding this comment

hyunsik commented Nov 19, 2015

blrunner commented Nov 19, 2015

hyunsik Nov 19, 2015

Choose a reason for hiding this comment

blrunner commented Nov 20, 2015

blrunner commented Nov 23, 2015

hyunsik Nov 24, 2015

Choose a reason for hiding this comment

blrunner commented Nov 24, 2015

blrunner commented Nov 25, 2015

hyunsik Dec 1, 2015

Choose a reason for hiding this comment

blrunner commented Dec 3, 2015

blrunner commented Dec 7, 2015

blrunner commented Dec 10, 2015

blrunner commented Jan 5, 2016

blrunner commented Jan 17, 2016

blrunner commented Mar 21, 2016