[SPARK-21322][SQL][followup] support histogram in filter cardinality estimation #19952
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloud-fan
commented
Dec 12, 2017
| * | ||
| * @param value a literal value of a column | ||
| * @param bins an array of bins for a given numeric equi-height histogram | ||
| * @return the id of the first bin into which a column value falls. |
There was a problem hiding this comment.
Now it's a private method and we can omit the parameter doc as it's trivial.
cloud-fan
commented
Dec 12, 2017
| * @param upperBound the highest value of the given range | ||
| * @param upperBoundInclusive whether the upperBound is included in the range | ||
| * @param lowerBound the lowest value of the given range | ||
| * @param lowerBoundInclusive whether the lowerBound is included in the range |
There was a problem hiding this comment.
instead of asking the callers to pass in the upper and lower bin id, it's more intuitive to pass whether to include the range boundaries.
|
Test build #84773 has finished for PR 19952 at commit
|
|
retest this please |
|
Test build #84784 has finished for PR 19952 at commit
|
wzhfy
reviewed
Dec 13, 2017
| * @param lowerId id of the low end bin holding the low end value of a column range | ||
| * @param higherEnd a given upper bound value of a specified column value range | ||
| * @param lowerEnd a given lower bound value of a specified column value range | ||
| * Note that the return value is double type, because the range boundaries usually occupy a |
| * @param lowerEnd a given lower bound value of a specified column value range | ||
| * Note that the return value is double type, because the range boundaries usually occupy a | ||
| * portion of a bin. An extrema case is [value, value] which is generated by equal predicate | ||
| * `col = value`, we can get more accuracy by allowing returning portion of histogram bins. |
| * @param min the lower bound of the current valid range for a given column | ||
| * @param datumNumber the numeric value of a literal | ||
| * @return the selectivity percentage for a condition in the current range. | ||
| * Computes the possibility of a equal predicate using histogram. |
| upperBound: Double, | ||
| upperBoundInclusive: Boolean, | ||
| lowerBound: Double, | ||
| lowerBoundInclusive: Boolean, | ||
| histogram: Histogram): Double = { |
There was a problem hiding this comment.
Is it better to pass the bin array instead of histogram? we can simplify many histogram.bins here.
| val numBinsHoldingEntireRange = EstimationUtils.numBinsHoldingRange( | ||
| max, upperBoundInclusive = true, min, lowerBoundInclusive = true, histogram) | ||
|
|
||
| val numBinsHoldingDatum = op match { |
ron8hu
suggested changes
Dec 13, 2017
| * [lowerBound, upperBound]. | ||
| * | ||
| * Note that the returned value is double type, because the range boundaries usually occupy a | ||
| * portion of a bin. An extrema case is [value, value] which is generated by equal predicate |
|
Test build #84822 has finished for PR 19952 at commit
|
|
Test build #84827 has finished for PR 19952 at commit
|
|
thanks, merging to master! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
some code cleanup/refactor and naming improvement.
How was this patch tested?
existing tests