-
Notifications
You must be signed in to change notification settings - Fork 12
Iterator predicate pushdown
Custom StorageHandler implementations can use the Hive API to analyze one or more predicates that appear in a query WHERE clause. The idea is that custom the Serde used in by the storage handler is better equipped to incorporate the predicate(s) into query restrictions for the underlying datastore, thus avoiding the need to have Hive do the much of the filtering.
The AccumuloStorageHandler supports decomposition of WHERE clause predicates involving (<, >, <=, >=, !=, =) over any of the defined Hive columns, including the rowID. Predicate conditions involving these operators are converted to one or more IteraterSettings that filter rows directly from the TabletServers.
The Hive API currently only allows predicate decomposition to the storage handler based for WHERE clauses that consist of purely conjuctive predicates.
Consider for example the query: SELECT * FROM table1 WHERE foo = 'hi' AND bar = 'there';
The above query consists of two predicates (foo = 'hi') and ('bar' = 'there'). Because they are combined into a single conjunctive restriction, they both are available to the StorageHandler.
In the case of SELECT * FROM table1 WHERE foo = 'hi' OR bar = 'there'; The Hive analyzer notices the disjunction, and neither of the predicates are available to the storage handler.
If even a singe disjunctive condition is found by the Hive predicate analyzer, the entire clause is evaluated at the Hive filter-level, and is not available for direct storage handler decomposition. So in the case of SELECT * FROM table1 WHERE foo = 'hi' AND bar = 'there' OR 'foo2' = 'whoa';, despite having two predicates ANDed together, the presence of the final OR operator means nothing can be passed down. This is a current limitation of the Hive API.
We can therefore assume predicates available to the AccumuloStorageHandler are meant to be ANDed together.
Predicates received by the storage handler are mapped 1-to-1 with a filter IteratorSetting. The Compare API in the storage handler makes use of a single class PrimitiveCompareFilter that extends WholeRowIterator and does all the filtering for a given qualifier.
As part of the Compare API, each IteratorSetting is supplied as part of the configuration options 4 critical parameters necessary to carry down the Hive column filtering to the TabletServers.
-
The column family and qualifier corresponding to the Hive column in question. (The 'age' in "age > 5")
-
The constant value being compared against. (The '5' in "age > 5").
-
An implementation of
PrimitiveCompare. Each instance is datatype specific and takes as an argument the constant value for comparison. The different methods encapsulate how to compare incoming byte[] values with the constant, and return either true or false. -
An implementation of
CompareOpt, which abstracts the comparison operation to hide the underlying data type. It takes an instance of PrimitiveCompare at construction time. (The '>' in "age > 5")
The iterator uses Reflection to dynamically build and instantiate the correct CompareOpt and PrimitiveCompare objects at query scan time.
Predicate pushdown is enabled by default. You can turn off iterator pushdown on a per-table basis by adding the property 'accumulo.no.iterators' = 'true' in the SERDEPROPERTIES for the table definition.
To use the custom iterator, make sure accumulo-hive-storage-handler-1.6.0-SNAPSHOT.jar is available on the classpath for each TabletServer. Otherwise queries involving WHERE clauses will produce errors.
An error will occur if a key/value pair is identified where the constant type being compared against does not the Hive column type.
Odd behavior can occur for key/value scans where matching qualifiers and column families across different rows correspond to heterogenous value types. For instance if row1 has an integer stored at cf and qual 'a', but row2 has a double for the equivalent cf and qual. Handling this is an ongoing work in progress.
One last thing worth mentioning. For predicates involving the Hive column mapped to rowID, the storage handler will attempt to do Range restrictions instead of Iterator filtering.
For example, the following SELECT
SELECT * from table1 where rowid >= "5555" will configure a single Range where the start row is greater than or equal to '5555', and the stop row is positive infinity. Multiple predicates involving the rowID will be converted to the equivalent number of Range restrictions.