Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
DRILL-2173: Partition queries to drive dynamic pruning
Adds new interface on the QueryContext as well as individual schemas for exploring partitions of tables. Adds injectable type for partition explorer for use in UDFs. This is hooked into both to expression materialization and interpreted evaluation. The FragmentContext throws an exception to tell users to turn on constant folding if a UDF that uses the PartitionExplorer makes it past planning. 2173 update -Address Chris' review comments. Change the PartitionExplorer to return an Iterable<String> instead of String[] Add interface level description to PartitionExplorer and StoragePluginPartitionExplorer. New inner class in FileSystemPlugin to fulfill the new Iterable interface for partitions. Formatting/cleanup fixes Clean up error reporting code in MaxDir UDF. Remove method to get a string from a DrillBuf, as it was already defined in StringFunctionHelpers. Add new utility method to specifically convert a VarCharHolder to a string to remove some boilerplate. Fixed an errant copy paste in a comment and removed unused imports. Fix docs in FileSystemPlugin, belongs with the 2173 changes. Fix references in Javadoc to properly use @link instead of @see. 2173 fixes, correctly return an empty list of sub-partitions if the path requested in the partition explorer interface is a file. Fix a few docs. More 2173, finishing Chris' comments 2173 update - Add validation for PartitionExplorer injectable in UdfUtiltiers. small change to fix refactored unit tests. cleanup for 2173 Fix maxdir UDF so it can compile in runtime generated code as well as the interpreted expression system (needed to fully qualify classes and interfaces). It still fails to execute, as we prevent requesting a schema from a non-root fragment. We do not expect these types of functions to ever be used without constant folding so this should not be an issue. Update error message in the case where the partition explorer is being used outside of planning. Adding free marker generated maxdir, imaxdir, mindir and imindir remove import that violates build checks, fix typo in new test class name Separate out SubDirectoryList from FileSystemSchemaFactory. Fix unit test to correctly test all four functions. Update partition explorer to take List instead of Collection. As the lists are used in parallel it should be explicit that these are expected to be ordered (which Collections do not guarantee). Drop the extra file generated due to the header in the free marker template and fix a typo and remove an unused import.
- Loading branch information
1 parent
ca73990
commit af7a52b
Showing
20 changed files
with
650 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
107 changes: 107 additions & 0 deletions
107
exec/java-exec/src/main/codegen/templates/DirectoryExplorers.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| /******************************************************************************* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| ******************************************************************************/ | ||
|
|
||
| <@pp.dropOutputFile /> | ||
|
|
||
| <@pp.changeOutputFile name="/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java" /> | ||
|
|
||
| <#include "/@includes/license.ftl" /> | ||
|
|
||
| package org.apache.drill.exec.expr.fn.impl; | ||
|
|
||
| import io.netty.buffer.DrillBuf; | ||
| import org.apache.drill.exec.expr.DrillSimpleFunc; | ||
| import org.apache.drill.exec.expr.annotations.FunctionTemplate; | ||
| import org.apache.drill.exec.expr.annotations.Output; | ||
| import org.apache.drill.exec.expr.annotations.Param; | ||
| import org.apache.drill.exec.expr.holders.VarCharHolder; | ||
|
|
||
| import javax.inject.Inject; | ||
|
|
||
| /** | ||
| * This file is generated with Freemarker using the template exec/java-exec/src/main/codegen/templates/DirectoryExplorers.java | ||
| */ | ||
| public class DirectoryExplorers { | ||
| static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(DirectoryExplorers.class); | ||
|
|
||
| <#list [ { "name" : "\"maxdir\"", "functionClassName" : "MaxDir", "comparison" : "compareTo(curr) < 0", "goal" : "maximum", "comparisonType" : "case-sensitive"}, | ||
| { "name" : "\"imaxdir\"", "functionClassName" : "IMaxDir", "comparison" : "compareToIgnoreCase(curr) < 0", "goal" : "maximum", "comparisonType" : "case-insensitive"}, | ||
| { "name" : "\"mindir\"", "functionClassName" : "MinDir", "comparison" : "compareTo(curr) > 0", "goal" : "minimum", "comparisonType" : "case-sensitive"}, | ||
| { "name" : "\"imindir\"", "functionClassName" : "IMinDir", "comparison" : "compareToIgnoreCase(curr) > 0", "goal" : "minimum", "comparisonType" : "case-insensitive"} | ||
| ] as dirAggrProps> | ||
|
|
||
|
|
||
| @FunctionTemplate(name = ${dirAggrProps.name}, scope = FunctionTemplate.FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.INTERNAL) | ||
| public static class ${dirAggrProps.functionClassName} implements DrillSimpleFunc { | ||
|
|
||
| @Param VarCharHolder schema; | ||
| @Param VarCharHolder table; | ||
| @Output VarCharHolder out; | ||
| @Inject DrillBuf buffer; | ||
| @Inject org.apache.drill.exec.store.PartitionExplorer partitionExplorer; | ||
|
|
||
| public void setup() { | ||
| } | ||
|
|
||
| public void eval() { | ||
| Iterable<String> subPartitions; | ||
| try { | ||
| subPartitions = partitionExplorer.getSubPartitions( | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(schema), | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(table), | ||
| new java.util.ArrayList<String>(), | ||
| new java.util.ArrayList<String>()); | ||
| } catch (org.apache.drill.exec.store.PartitionNotFoundException e) { | ||
| throw new RuntimeException( | ||
| String.format("Error in %s function: Table %s does not exist in schema %s ", | ||
| ${dirAggrProps.name}, | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(table), | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(schema)) | ||
| ); | ||
| } | ||
| java.util.Iterator partitionIterator = subPartitions.iterator(); | ||
| if (!partitionIterator.hasNext()) { | ||
| throw new RuntimeException( | ||
| String.format("Error in %s function: Table %s in schema %s does not contain sub-partitions.", | ||
| ${dirAggrProps.name}, | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(table), | ||
| org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(schema) | ||
| ) | ||
| ); | ||
| } | ||
| String subPartitionStr = (String) partitionIterator.next(); | ||
| String curr; | ||
| // find the ${dirAggrProps.goal} directory in the list using a ${dirAggrProps.comparisonType} string comparison | ||
| while (partitionIterator.hasNext()){ | ||
| curr = (String) partitionIterator.next(); | ||
| if (subPartitionStr.${dirAggrProps.comparison}) { | ||
| subPartitionStr = curr; | ||
| } | ||
| } | ||
| String[] subPartitionParts = subPartitionStr.split(java.io.File.separator); | ||
| subPartitionStr = subPartitionParts[subPartitionParts.length - 1]; | ||
| byte[] result = subPartitionStr.getBytes(); | ||
| out.buffer = buffer = buffer.reallocIfNeeded(result.length); | ||
|
|
||
| out.buffer.setBytes(0, subPartitionStr.getBytes(), 0, result.length); | ||
| out.start = 0; | ||
| out.end = result.length; | ||
| } | ||
| } | ||
| </#list> | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.