Skip to content

Commit

Permalink
[SPARK-8841] [SQL] Fix partition pruning percentage log message
Browse files Browse the repository at this point in the history
When pruning partitions for a query plan, a message is logged indicating what how many partitions were selected based on predicate criteria, and what percent were pruned.

The current release erroneously uses `1 - total/selected` to compute this quantity, leading to nonsense messages like "pruned -1000% partitions". The fix is simple and obvious.

Author: Steve Lindemann <steve.lindemann@engineersgatelp.com>

Closes #7227 from srlindemann/master and squashes the following commits:

c788061 [Steve Lindemann] fix percentPruned log message
  • Loading branch information
eglp-slindemann authored and liancheng committed Jul 6, 2015
1 parent 86768b7 commit 39e4e7e
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ private[sql] object DataSourceStrategy extends Strategy with Logging {
logInfo {
val total = t.partitionSpec.partitions.length
val selected = selectedPartitions.length
val percentPruned = (1 - total.toDouble / selected.toDouble) * 100
val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
s"Selected $selected partitions out of $total, pruned $percentPruned% partitions."
}

Expand Down

0 comments on commit 39e4e7e

Please sign in to comment.