Skip to content

Commit

Permalink
[SPARK-8680] [SQL] Slightly improve PropagateTypes
Browse files Browse the repository at this point in the history
JIRA: https://issues.apache.org/jira/browse/SPARK-8680

This PR slightly improve `PropagateTypes` in `HiveTypeCoercion`. It moves `q.inputSet` outside `q transformExpressions` instead calling `inputSet` multiple times. It also builds a map of attributes for looking attribute easily.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #7087 from viirya/improve_propagatetypes and squashes the following commits:

5c314c1 [Liang-Chi Hsieh] For comments.
913f6ad [Liang-Chi Hsieh] Slightly improve PropagateTypes.
  • Loading branch information
viirya authored and Davies Liu committed Jun 30, 2015
1 parent 865a834 commit a48e619
Showing 1 changed file with 16 additions and 14 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -131,20 +131,22 @@ trait HiveTypeCoercion {
// Don't propagate types from unresolved children.
case q: LogicalPlan if !q.childrenResolved => q

case q: LogicalPlan => q transformExpressions {
case a: AttributeReference =>
q.inputSet.find(_.exprId == a.exprId) match {
// This can happen when a Attribute reference is born in a non-leaf node, for example
// due to a call to an external script like in the Transform operator.
// TODO: Perhaps those should actually be aliases?
case None => a
// Leave the same if the dataTypes match.
case Some(newType) if a.dataType == newType.dataType => a
case Some(newType) =>
logDebug(s"Promoting $a to $newType in ${q.simpleString}}")
newType
}
}
case q: LogicalPlan =>
val inputMap = q.inputSet.toSeq.map(a => (a.exprId, a)).toMap
q transformExpressions {
case a: AttributeReference =>
inputMap.get(a.exprId) match {
// This can happen when a Attribute reference is born in a non-leaf node, for example
// due to a call to an external script like in the Transform operator.
// TODO: Perhaps those should actually be aliases?
case None => a
// Leave the same if the dataTypes match.
case Some(newType) if a.dataType == newType.dataType => a
case Some(newType) =>
logDebug(s"Promoting $a to $newType in ${q.simpleString}}")
newType
}
}
}
}

Expand Down

0 comments on commit a48e619

Please sign in to comment.