Skip to content

[SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion#57327

Draft
jiwen624 wants to merge 1 commit into
apache:masterfrom
jiwen624:SPARK-58164
Draft

[SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion#57327
jiwen624 wants to merge 1 commit into
apache:masterfrom
jiwen624:SPARK-58164

Conversation

@jiwen624

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

approx_top_k_estimate picked its sketch deserialization SerDe from the static item type in the sketch-state struct (field 2). A UNION ALL of two sketch columns with coercible item types (e.g. INT + BIGINT) widens that static type while the serialized bytes and the per-row DDL (field 3) keep the original type. Estimation then decoded the bytes with the wrong SerDe.

This PR makes estimate deserialize using the true item type recorded per-row in the state DDL (recovering collation from the static type, exactly as combine already does), then widens each decoded item to the declared output type via a lossless cast so results still match the statically-declared schema.

Why are the changes needed?

A query like:

SELECT approx_top_k_estimate(approx_top_k_combine(sketch, 5))
FROM (SELECT approx_top_k_accumulate(c) AS sketch, 'int' AS tag
        FROM VALUES (1), (1), (2) AS t(c)
      UNION ALL
      SELECT approx_top_k_accumulate(c) AS sketch, 'long' AS tag
        FROM VALUES (CAST(9 AS BIGINT)) AS t(c))
WHERE tag = 'int';

fails at runtime with SketchesArgumentException: Bounds Violation, e.g.,

26/07/16 22:29:33 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
org.apache.datasketches.common.SketchesArgumentException: Bounds Violation: reqOffset: 0, reqLength: 16, (reqOff + reqLen): 16, allocSize: 10
        at org.apache.datasketches.common.Util.checkBounds(Util.java:659)

Or, depending on the widening it can also throw ClassCastException or, when byte sizes
happen to match (e.g. BIGINT→DOUBLE), silently return wrong results.

Does this PR introduce any user-facing change?

Yes. A query that may crash or return wrong results now gets fixed.

How was this patch tested?

Added UT.

Was this patch authored or co-authored using generative AI tooling?

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant