[SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion#57327
Draft
jiwen624 wants to merge 1 commit into
Draft
Conversation
…ype was widened by type coercion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
approx_top_k_estimatepicked its sketch deserialization SerDe from the static item type in the sketch-state struct (field 2). AUNION ALLof two sketch columns with coercible item types (e.g. INT + BIGINT) widens that static type while the serialized bytes and the per-row DDL (field 3) keep the original type. Estimation then decoded the bytes with the wrong SerDe.This PR makes
estimatedeserialize using the true item type recorded per-row in the state DDL (recovering collation from the static type, exactly ascombinealready does), then widens each decoded item to the declared output type via a lossless cast so results still match the statically-declared schema.Why are the changes needed?
A query like:
fails at runtime with SketchesArgumentException: Bounds Violation, e.g.,
Or, depending on the widening it can also throw ClassCastException or, when byte sizes
happen to match (e.g. BIGINT→DOUBLE), silently return wrong results.
Does this PR introduce any user-facing change?
Yes. A query that may crash or return wrong results now gets fixed.
How was this patch tested?
Added UT.
Was this patch authored or co-authored using generative AI tooling?
Yes