Skip to content

[fix](move-memtable) check segment id in add_segment#29898

Merged
dataroaring merged 2 commits intoapache:masterfrom
kaijchen:mm-add-segment-check
Jan 12, 2024
Merged

[fix](move-memtable) check segment id in add_segment#29898
dataroaring merged 2 commits intoapache:masterfrom
kaijchen:mm-add-segment-check

Conversation

@kaijchen
Copy link
Member

Proposed changes

Check (src_id, segid) mapping and return error if not found.
Avoid throwing out of range exception.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38708 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9, data reload: false

------ Round 1 ----------------------------------
q1	17950	5008	4977	4977
q2	2042	145	133	133
q3	10684	1133	1216	1133
q4	10238	826	842	826
q5	7975	3176	3194	3176
q6	204	124	123	123
q7	895	513	491	491
q8	9265	1951	1964	1951
q9	7386	6483	6459	6459
q10	8265	3116	3061	3061
q11	430	220	212	212
q12	363	202	199	199
q13	18109	3431	3445	3431
q14	248	212	238	212
q15	562	528	505	505
q16	437	380	378	378
q17	942	475	497	475
q18	7412	6745	6653	6653
q19	1597	1342	1414	1342
q20	571	294	299	294
q21	2775	2371	2443	2371
q22	347	306	313	306
Total cold run time: 108697 ms
Total hot run time: 38708 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4978	4971	4954	4954
q2	319	206	207	206
q3	3359	3329	3285	3285
q4	2254	2246	2260	2246
q5	5826	5823	5799	5799
q6	195	120	119	119
q7	2324	1839	1866	1839
q8	3466	3551	3561	3551
q9	8820	8868	8758	8758
q10	3749	3825	3841	3825
q11	585	437	431	431
q12	802	602	603	602
q13	5795	3227	3204	3204
q14	288	248	271	248
q15	562	518	513	513
q16	505	444	455	444
q17	2056	2014	2025	2014
q18	8793	8486	8334	8334
q19	1600	1565	1598	1565
q20	2168	1938	1928	1928
q21	5902	5687	5746	5687
q22	585	471	484	471
Total cold run time: 64931 ms
Total hot run time: 60023 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.71% (8650/23565)
Line Coverage: 28.72% (70449/245283)
Region Coverage: 27.64% (36400/131715)
Branch Coverage: 24.36% (18626/76476)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9_4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9/report/index.html

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 12, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s
storage size: 17183831053 Bytes

@doris-robot
Copy link

TPC-DS: Total hot run time: 179478 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9, data reload: false

query1	932	332	331	331
query2	6718	1823	1913	1823
query3	6712	206	203	203
query4	26080	22127	22134	22127
query5	5566	525	504	504
query6	268	191	188	188
query7	4618	295	280	280
query8	234	196	196	196
query9	9155	2735	2840	2735
query10	497	238	242	238
query11	16249	15350	15222	15222
query12	127	71	70	70
query13	1695	403	386	386
query14	11761	7160	7207	7160
query15	242	193	198	193
query16	6501	249	243	243
query17	1951	491	481	481
query18	1940	267	261	261
query19	286	145	146	145
query20	80	79	71	71
query21	200	129	118	118
query22	4896	4889	4750	4750
query23	31794	30895	30897	30895
query24	11115	2835	2838	2835
query25	562	329	326	326
query26	1413	152	154	152
query27	2776	286	277	277
query28	6941	1871	1851	1851
query29	1121	693	659	659
query30	282	138	141	138
query31	985	754	775	754
query32	94	60	59	59
query33	712	247	241	241
query34	1011	466	471	466
query35	875	775	731	731
query36	1285	1238	1261	1238
query37	100	68	69	68
query38	3376	3298	3266	3266
query39	1355	1297	1292	1292
query40	214	102	91	91
query41	41	36	35	35
query42	106	94	100	94
query43	513	516	527	516
query44	1071	701	709	701
query45	207	187	180	180
query46	1052	649	672	649
query47	1623	1555	1594	1555
query48	414	320	325	320
query49	1126	330	314	314
query50	725	315	324	315
query51	5361	5313	5330	5313
query52	103	88	98	88
query53	355	302	288	288
query54	2055	1574	1600	1574
query55	101	87	86	86
query56	217	198	194	194
query57	1007	946	952	946
query58	221	202	201	201
query59	2728	2597	2532	2532
query60	241	217	238	217
query61	84	83	81	81
query62	593	385	384	384
query63	320	288	290	288
query64	4789	1688	1732	1688
query65	3366	3270	3266	3266
query66	1046	325	330	325
query67	15287	15417	15203	15203
query68	13076	507	507	507
query69	643	381	385	381
query70	1722	1432	1376	1376
query71	500	232	242	232
query72	5020	2843	2827	2827
query73	3247	320	307	307
query74	6975	6405	6420	6405
query75	5495	2303	2310	2303
query76	6435	1062	1125	1062
query77	709	266	269	266
query78	9943	8717	8583	8583
query79	1075	504	487	487
query80	559	335	343	335
query81	465	204	199	199
query82	207	87	87	87
query83	168	124	118	118
query84	272	74	76	74
query85	1021	363	329	329
query86	414	382	366	366
query87	3555	3445	3394	3394
query88	3149	2203	2202	2202
query89	464	404	401	401
query90	1836	207	201	201
query91	183	135	137	135
query92	63	53	57	53
query93	1373	396	429	396
query94	1304	181	182	181
query95	538	483	482	482
query96	623	317	316	316
query97	4267	4202	4206	4202
query98	223	189	186	186
query99	998	721	668	668
Total cold run time: 293993 ms
Total hot run time: 179478 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9, data reload: false

query1	0.05	0.06	0.04
query2	0.06	0.03	0.03
query3	0.25	0.11	0.12
query4	1.76	0.12	0.11
query5	0.53	0.51	0.53
query6	1.39	0.65	0.63
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.57	0.50	0.50
query10	0.55	0.56	0.56
query11	0.14	0.09	0.09
query12	0.12	0.09	0.08
query13	0.61	0.60	0.60
query14	0.79	0.78	0.82
query15	0.80	0.80	0.79
query16	0.34	0.34	0.35
query17	0.97	0.96	0.98
query18	0.25	0.24	0.25
query19	1.85	1.76	1.70
query20	0.01	0.01	0.01
query21	15.39	0.54	0.55
query22	2.29	2.40	2.09
query23	17.29	0.92	0.80
query24	16.20	0.57	0.59
query25	2.16	0.15	0.14
query26	0.13	0.14	0.13
query27	0.15	0.15	0.15
query28	6.56	0.84	0.79
query29	12.55	3.23	3.29
query30	0.57	0.48	0.48
query31	2.78	0.35	0.36
query32	3.37	0.48	0.48
query33	3.18	3.24	3.19
query34	15.77	4.19	4.21
query35	4.18	4.21	4.20
query36	1.11	1.05	1.05
query37	0.07	0.05	0.05
query38	0.03	0.02	0.03
query39	0.02	0.02	0.02
query40	0.15	0.13	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 115.17 s
Total hot run time: 30.77 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4cc3f09a53130a069c6f2d8baf4e4d727a7bd3b9 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       12.6 seconds inserted 10000000 Rows, about 793K ops/s

Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants