-
Notifications
You must be signed in to change notification settings - Fork 12
Iterator predicate pushdown
Custom StorageHandler implementations can use the Hive API to analyze one or more predicates appearing in query WHERE clauses for direct. The idea is that custom Serde used in the storage handler is better equipped to incorporate the predicate(s) into query restrictions for the underlying datastore, thus avoiding the need to Hive filter at higher level.
The AccumuloStorageHandler supports decomposition of WHERE clause predicates involving <, >, <=, >=, !=, and = over any of the defined Hive columns, including the rowID. Predicate conditions involving these operators are converted to one or more IteraterSettings that filter rows directly from the TabletServers.
The Hive API currently only allows predicate decomposition to the storage handler based for WHERE clauses that consist of purely conjuctive predicates.
For example:
SELECT * FROM table1 WHERE foo = 'hi' AND bar = 'there';
The above query consists of two predicates (foo = 'hi') and ('bar' = 'there'). Because they are combined into a single conjunctive restriction, they both are available to the StorageHandler.
In the case of SELECT * FROM table1 WHERE foo = 'hi' OR bar = 'there'; Neither of the predicates are available to the storage handler.
If even a singe disjunctive condition is found by the Hive predicate analyzer, the entire clause is evaluated at the Hive filter-level, and is not available for direct storage handler decomposition. So in the case of SELECT * FROM table1 WHERE foo = 'hi' AND bar = 'there' OR 'foo2' = 'whoa';, despite having two predicates ANDed together, the presence of the final OR operator means nothing can be passed down. This is a currently limitation with the Hive API.
We can naively assume predicates sent to the Accumulo storage handler are meant to be ANDed together else they wouldn't be visible to us in the first place.
Predicates received by the storage handler are mapped 1-to-1 with a filter IteratorSetting. The Compare API in the AccumuloStorageHandler makes use of a single class PrimitiveCompareFilter that extends WholeRowIterator.
As part of the Compare API, each IteratorSetting is supplied as part of the configuration options 4 critical parameters necessary to carry down the Hive column filtering to the TabletServers.
-
The column family and qualifier corresponding to the Hive column in question. (The 'age' in "age > 5")
-
The constant value being compared against. (The '5' in "age > 5").
-
An implementation of
PrimitiveCompare. Each instance is datatype specific and takes in the constant value for comparison at construction time. The different methods encapsulates how to compare byte[] values against the constant. -
An implementation of
CompareOpt, which abstracts the comparison operation to hide the underlying data type. It takes an instance of PrimitiveCompare at construction time. (The '>' in "age > 5")
The iterator uses Reflection to dynamically build and instantiate each CompareOpt and PrimitiveCompare objects at query time.
Predicate pushdown is enabled by default. You can turn off iterator pushdown on a per-table basis by adding the property 'accumulo.no.iterators' = 'true' in the SERDEPROPERTIES for the table definition.
To use the custom iterator, make sure the accumulo-hive-storage-handler jar is available on the classpath for each TabletServer. Otherwise the turns
An error will occur if a key/value pair is identified where the constant type being compared against does not the Hive column type.
Odd behavior can occur for key/value scans where matching qualifiers and column families across different rows correspond to heterogenous value types. For instance if row1 has an integer stored at cf and qual 'a', but row2 has a double for the equivalent cf and qual. Handling this is an ongoing work in progress.
One last thing worth mentioning. For predicates involving the Hive column mapped to RowID, the storage handler will attempt to do Range restrictions instead of Iterator filtering.