test: docs and test-coverage hardening for native collect_list / collect_set - #5055
Open
andygrove wants to merge 2 commits into
Open
test: docs and test-coverage hardening for native collect_list / collect_set#5055andygrove wants to merge 2 commits into
andygrove wants to merge 2 commits into
Conversation
Follow-up to PR apache#4720. Adds the missing `collect_set` audit entry, covers the split-execution cascade and the multi-stage fallback guard with assertions, and extends the SQL fixture datatype coverage. - docs: add a `## collect_set` section to the agg_funcs audit page and link the `collect_list` entry to it instead of referring to it in prose - test: add mixed-engine `collect_list` tests for Comet partial + Spark final and Spark partial + Comet final, exercising `hasNativeArrayBufferAgg` and `tagUnsafePartialAggregates` - test: assert the exact fallback reason in the distinct-combined test instead of only checking the answer - test: add `timestamp_ntz` (native path) and ANSI interval (Spark fallback) blocks to the `collect_list` SQL fixture
- test: assert on short discriminating fallback-reason fragments instead of copying the full message text from CometExecRule / operators - test: collapse the two mixed-engine tests into one parameterized pair and pin the Comet aggregate count at zero, so they fail if the partial/final cascade regresses instead of passing on any plan shape - test: drop the dead `ORDER BY`, which the answer helper already handles - test: fold the timestamp_ntz fixture into the existing timestamp table rather than duplicating its rows, matching the file's convention - test: pin the interval fallback with `expect_fallback` instead of `spark_answer_only`, so the block comment's claim is enforced - docs: move the shared buffer-shape note under collect_list, correct the `adjustOutputForNativeState` owner to CometBaseAggregate, and drop the redundant containsNull and Spark 4.2 restatements
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5024.
Rationale for this change
PR #4720 landed the native
collect_list/array_aggimplementation and deferred four docs and test-coverage items to a follow-on so they would not block the feature. This PR addresses those items.Two of them close real gaps in the regression net: neither
collect_listtest toggled the partial/final hash aggregate configs, so thehasNativeArrayBufferAgg+tagUnsafePartialAggregatessplit-execution path was untested; and the distinct-combined test only compared answers, so it would have passed even if the #4724 guard stopped firing and the query silently ran natively.What changes are included in this PR?
collect_setaudit entry.docs/source/contributor-guide/expression-audits/agg_funcs.mddocumentedcollect_listbut notcollect_set, even though feat: native collect_list / array_agg aggregate #4720 changedCometCollectSetsemantically. Adds a## collect_setsection covering Spark 3.4.3 / 3.5.8 / 4.0.1 / 4.1.1, theSparkCollectSetdelegation, theBinaryType-vs-list buffer-shape mismatch, and theNaN-dedupIncompatiblereason on float/double. Thecollect_listentry now links to it instead of referring to it in prose.Split-execution cascade regression tests. Two new tests in
CometAggregateSuite, mirroring the existingmixed engine sum/avgpair:mixed engine collect_list: Comet partial + Spark final matches Spark(COMET_ENABLE_FINAL_HASH_AGGREGATE=false)mixed engine collect_list: Spark partial + Comet final matches Spark(COMET_ENABLE_PARTIAL_HASH_AGGREGATE=false), the case whereadjustOutputForNativeStatewould misinterpret Spark'sBinaryTypebuffer as a list if the final ran natively.Fallback-reason assertions in the distinct-combined test.
checkSparkAnsweris replaced withcheckSparkAnswerAndFallbackReason(s). The two branches are protected by different guards, so each asserts its own:tagUnsafePartialAggregatesdoes not fire and the dependent stages fall back via the buffer-source check (Comet aggregate that merges intermediate buffers requires a Comet child aggregate ... Incompatible aggregate function(s): collect_list).Partial aggregate disabled: part of a multi-stage CollectList/CollectSet aggregate whose intermediate buffer cannot round-trip in Comet (issue #4724)) is asserted alongside the buffer-source reason.SQL fixture datatype coverage.
collect_list.sqlgains atimestamp_ntzblock (inQueryPlanSerde.supportedDataType, so it runs through the nativeSparkCollectListaccumulator, previously untested) and a year-month / day-time ANSI interval block (not supported natively, sospark_answer_onlycovers the Spark fallback). NoMinSparkVersiongate is needed: both work on 3.4, which is already the file's floor.How are these changes tested?
CometAggregateSuiteandCometSqlFileTestSuitewere run against both ends of the supported range, Spark 3.4 and the default Spark 4.1, and pass on both:The
NaN/-0.0deduplication behavior asserted in the new audit entry was verified against Spark directly rather than inferred:collect_setover(0.0, -0.0, NaN, NaN, 1.0)returns[0.0, 1.0, NaN, NaN], confirming Spark keeps eachNaNwhile collapsing+0.0and-0.0.