Skip to content

[fix](variant) Apply sparse/doc mixed-row review fixes#65957

Draft
hello-stephen wants to merge 1 commit into
apache:masterfrom
hello-stephen:agent/fix-variant-review-followup
Draft

[fix](variant) Apply sparse/doc mixed-row review fixes#65957
hello-stephen wants to merge 1 commit into
apache:masterfrom
hello-stephen:agent/fix-variant-review-followup

Conversation

@hello-stephen

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #65660

Problem Summary:

PR #65660 was merged before its separate review follow-up could land. Two correctness items therefore remain on master:

  • doc-value offsets are cumulative, so checking a non-zero end offset can let a previous row's doc snapshot hide sparse fields in the current row;
  • two empty-object rows in the regression oracle were rendered as arrays, which would lock unrelated corruption into the expected output.

This PR reapplies only the four-file review follow-up on top of current master. It uses the current-row doc-value offset delta, adds the prior-doc/current-sparse unit scenario, and removes the two unrelated oracle rows.

Release note

None

Check List (For Author)

  • Test:
    • build-support/clang-format.sh and build-support/check-format.sh with clang-format 16 passed.
    • git diff --check passed.
    • The changed Groovy case compiled with Groovy 4.0.19 and was discovered as test_topn_lazy_materialize_sparse_variant in group p0.
    • Full local regression framework packaging was attempted but stopped before source compilation because the worktree dependency bundle does not contain thirdparty/installed/bin/thrift; BE UT and buildall are required in CI.
  • Behavior changed: Yes. Current-row sparse fields are no longer hidden by a previous row's doc snapshot.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65660

Problem Summary: The merged sparse VARIANT visibility fix still used a cumulative doc-value end offset, so doc data from an earlier row could hide sparse fields in the current row. Its regression oracle also retained two rows whose empty objects were rendered as arrays. Use the current-row doc-value offset delta, add the prior-doc/current-sparse unit scenario, and remove the unrelated bad oracle rows.

### Release note

None

### Check List (For Author)

- Test: clang-format 16 check, git diff --check, and Groovy syntax/discovery check passed. BE UT and buildall are required in CI.
- Behavior changed: Yes. Current-row sparse fields are no longer hidden by a previous row doc snapshot.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor Author

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 28th, 2026 5:08 PM.
Workflow run: https://github.com/apache/doris/actions/runs/29992772889

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor Author
TPC-H: Total hot run time: 29615 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8eff1c079c191871daa1465f30474b68783f20f5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17732	4179	4238	4179
q2	1988	318	209	209
q3	10307	1420	844	844
q4	4711	475	336	336
q5	7615	850	574	574
q6	207	179	143	143
q7	752	812	613	613
q8	10209	1590	1610	1590
q9	5947	4287	4293	4287
q10	6805	1726	1444	1444
q11	507	358	326	326
q12	749	590	466	466
q13	18090	3305	2723	2723
q14	272	260	253	253
q15	q16	795	776	719	719
q17	1040	918	1129	918
q18	6901	5706	5561	5561
q19	1449	1389	1070	1070
q20	823	681	600	600
q21	5659	2691	2457	2457
q22	438	367	303	303
Total cold run time: 102996 ms
Total hot run time: 29615 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4586	4507	4479	4479
q2	289	327	215	215
q3	4628	4992	4428	4428
q4	2075	2164	1387	1387
q5	4468	4334	4314	4314
q6	244	184	131	131
q7	2313	1968	1587	1587
q8	2468	2274	2193	2193
q9	7765	7794	7723	7723
q10	4668	4667	4169	4169
q11	598	658	400	400
q12	766	761	578	578
q13	3277	3587	3027	3027
q14	303	314	276	276
q15	q16	699	751	628	628
q17	1399	1368	1366	1366
q18	8266	7509	6797	6797
q19	1125	1073	1121	1073
q20	2216	2209	1946	1946
q21	5311	4679	4535	4535
q22	523	468	408	408
Total cold run time: 57987 ms
Total hot run time: 51660 ms

@hello-stephen

Copy link
Copy Markdown
Contributor Author
TPC-DS: Total hot run time: 178184 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8eff1c079c191871daa1465f30474b68783f20f5, data reload: false

query5	4347	641	470	470
query6	472	229	206	206
query7	4876	624	341	341
query8	335	190	175	175
query9	8748	4040	4060	4040
query10	484	377	314	314
query11	5911	2322	2116	2116
query12	155	102	101	101
query13	1280	642	457	457
query14	6399	5242	4919	4919
query14_1	4308	4302	4326	4302
query15	217	209	182	182
query16	1026	463	453	453
query17	1130	727	603	603
query18	2670	487	357	357
query19	218	199	157	157
query20	115	106	107	106
query21	234	166	143	143
query22	13599	13581	13325	13325
query23	17161	16378	16060	16060
query23_1	16157	16262	16210	16210
query24	7432	1795	1313	1313
query24_1	1322	1321	1289	1289
query25	565	461	397	397
query26	1346	342	221	221
query27	2659	633	385	385
query28	4507	2067	2030	2030
query29	1103	661	499	499
query30	337	266	223	223
query31	1112	1099	990	990
query32	109	65	62	62
query33	529	330	265	265
query34	1186	1127	650	650
query35	770	785	668	668
query36	1162	1167	1068	1068
query37	161	112	97	97
query38	1880	1706	1668	1668
query39	886	885	847	847
query39_1	834	836	861	836
query40	262	205	138	138
query41	65	63	61	61
query42	94	90	91	90
query43	332	337	286	286
query44	1473	790	762	762
query45	190	190	183	183
query46	1114	1244	718	718
query47	2117	2111	1963	1963
query48	410	400	281	281
query49	570	416	311	311
query50	1101	437	328	328
query51	10666	10685	10573	10573
query52	84	86	74	74
query53	253	286	205	205
query54	278	239	220	220
query55	73	70	67	67
query56	290	298	288	288
query57	1303	1274	1169	1169
query58	279	265	262	262
query59	1637	1656	1438	1438
query60	312	270	264	264
query61	148	153	149	149
query62	538	494	431	431
query63	239	215	209	209
query64	2807	1043	926	926
query65	4729	4659	4636	4636
query66	1781	506	387	387
query67	29210	29122	29078	29078
query68	3137	1592	1017	1017
query69	415	304	257	257
query70	1078	957	952	952
query71	391	335	357	335
query72	3080	2725	2399	2399
query73	846	841	434	434
query74	5060	4933	4740	4740
query75	2551	2509	2124	2124
query76	2344	1197	791	791
query77	357	369	297	297
query78	11879	11904	11343	11343
query79	1445	1200	761	761
query80	1319	558	487	487
query81	557	335	294	294
query82	608	161	119	119
query83	364	327	313	313
query84	286	155	132	132
query85	981	612	517	517
query86	431	287	268	268
query87	1833	1819	1734	1734
query88	3745	2798	2790	2790
query89	441	375	327	327
query90	1925	205	202	202
query91	206	197	168	168
query92	59	61	52	52
query93	1758	1436	979	979
query94	739	367	330	330
query95	799	578	489	489
query96	1082	806	336	336
query97	2635	2616	2476	2476
query98	212	200	200	200
query99	1100	1108	963	963
Total cold run time: 263986 ms
Total hot run time: 178184 ms

@hello-stephen

Copy link
Copy Markdown
Contributor Author
ClickBench: Total hot run time: 25.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8eff1c079c191871daa1465f30474b68783f20f5, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.04
query3	0.25	0.13	0.13
query4	1.61	0.17	0.13
query5	0.24	0.23	0.22
query6	1.24	1.07	1.04
query7	0.03	0.00	0.00
query8	0.06	0.04	0.03
query9	0.37	0.32	0.31
query10	0.54	0.56	0.54
query11	0.19	0.14	0.14
query12	0.19	0.14	0.14
query13	0.46	0.47	0.48
query14	1.02	1.02	1.01
query15	0.60	0.60	0.59
query16	0.31	0.31	0.34
query17	1.12	1.07	1.11
query18	0.22	0.20	0.20
query19	2.07	1.97	1.97
query20	0.02	0.01	0.01
query21	15.40	0.24	0.15
query22	4.83	0.06	0.05
query23	16.12	0.30	0.12
query24	2.97	0.43	0.32
query25	0.11	0.04	0.03
query26	0.74	0.19	0.16
query27	0.04	0.04	0.04
query28	3.47	0.94	0.55
query29	12.55	4.01	3.30
query30	0.28	0.16	0.16
query31	2.76	0.58	0.31
query32	3.22	0.58	0.48
query33	3.23	3.19	3.18
query34	15.46	4.25	3.54
query35	3.52	3.50	3.54
query36	0.54	0.44	0.42
query37	0.09	0.06	0.06
query38	0.06	0.03	0.03
query39	0.04	0.03	0.03
query40	0.20	0.17	0.15
query41	0.07	0.03	0.02
query42	0.03	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 96.42 s
Total hot run time: 25.02 s

@hello-stephen

Copy link
Copy Markdown
Contributor Author

[nonconcurrent-1003981-shared-agent-failure]

NonConcurrent build #1003981 is terminal FAILURE, but its 51 failed cases are one startup cascade rather than independent case failures.

First actionable error in the uploaded BE log occurs before any case runs: the BE bundled Python environment cannot be created because ensurepip is unavailable and the agent asks for python3.8-venv. The BE then exits; FE reports backend 1784804751107 as not alive, so all 51 cases fail with No backend available as scan node. The later Stop grace fail and NO NEW FE IMAGE FILE CREATED AFTER FE RESTART are cleanup/checkpoint consequences.

This build ran on agent 20260723185329agent_172.17.0.72_1; other same-wave NonConcurrent builds on other agents progressed into passing cases. No case-level or #65957-specific code failure is established from #1003981.

Please rerun NonConcurrent on a clean agent (or repair the missing Python venv dependency on this agent) and use that terminal result for the PR gate.

@hello-stephen

Copy link
Copy Markdown
Contributor Author

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.14% (30979/41231)
Line Coverage 59.59% (344203/577610)
Region Coverage 56.29% (289171/513721)
Branch Coverage 57.59% (128998/224000)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant