[fix](partial update) compaction may cause update failue

[zhannngchen]

Proposed changes

Issue Number: close #xxx

Under normal circumstances, get_rowset_by_ids does not need to consider the stale rowset, in other word, if a rowset id is not found in the normal rowset, we can ignore it. This is because even if we handle stale rowset here, we need to recalculate the new rowset generated by the corresponding compaction in the publish phase.

However, for partial update, ignoring the stale rowset may cause some keys to not be found in the flush phase (lookup_row_key returns KEY_NOT_FOUND), and thus be mistaken as new keys in the flush phase, which will cause the load to fail in the following two cases:

1. when strict_mode is enabled, new keys are not allowed to be added.
2. Some columns that need to be filled are neither nullable nor have a default value, in which case the value of the field cannot be filled as a new key, leading to a failure of the load.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

[zhannngchen]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[zhannngchen]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TPC-H: Total hot run time: 37823 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a8d67e04fa3b8ea62a848304117eb294c55298c, data reload: false

------ Round 1 ----------------------------------
q1	17726	4052	4028	4028
q2	2066	145	134	134
q3	10570	949	939	939
q4	4659	946	953	946
q5	7615	2851	2999	2851
q6	182	121	124	121
q7	1223	756	754	754
q8	9340	2022	2034	2022
q9	7219	6294	6300	6294
q10	8218	2494	2455	2455
q11	411	218	192	192
q12	702	314	318	314
q13	17950	3239	3230	3230
q14	291	250	277	250
q15	531	500	494	494
q16	481	394	409	394
q17	937	832	795	795
q18	6921	5979	6118	5979
q19	1561	1516	1513	1513
q20	547	286	273	273
q21	6696	3560	3607	3560
q22	818	285	285	285
Total cold run time: 106664 ms
Total hot run time: 37823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4008	3998	4021	3998
q2	301	211	214	211
q3	2978	2929	2951	2929
q4	1857	1832	1845	1832
q5	5166	5122	5163	5122
q6	201	115	114	114
q7	2123	1699	1668	1668
q8	3119	3201	3212	3201
q9	8239	8272	8233	8233
q10	6077	3667	3650	3650
q11	495	435	432	432
q12	697	509	555	509
q13	10562	3064	3044	3044
q14	263	247	259	247
q15	537	494	489	489
q16	509	465	479	465
q17	1709	1716	1684	1684
q18	7838	7474	7436	7436
q19	1654	1637	1642	1637
q20	2093	1911	1886	1886
q21	4894	4574	4618	4574
q22	515	442	437	437
Total cold run time: 65835 ms
Total hot run time: 53798 ms

[doris-robot]

TPC-DS: Total hot run time: 169099 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a8d67e04fa3b8ea62a848304117eb294c55298c, data reload: false

query1	927	348	332	332
query2	6545	1777	1781	1777
query3	6706	211	202	202
query4	23262	20583	20526	20526
query5	4274	365	356	356
query6	267	172	187	172
query7	4631	307	303	303
query8	262	192	195	192
query9	8545	2202	2192	2192
query10	424	215	216	215
query11	14519	14201	14023	14023
query12	143	84	85	84
query13	1645	432	444	432
query14	8331	6530	6192	6192
query15	206	172	172	172
query16	7087	268	254	254
query17	1027	518	502	502
query18	1911	281	268	268
query19	192	148	154	148
query20	86	82	81	81
query21	197	131	119	119
query22	4771	4689	4691	4689
query23	30541	29861	29902	29861
query24	11975	3138	3105	3105
query25	657	345	358	345
query26	1913	158	162	158
query27	3054	325	324	324
query28	6992	1793	1785	1785
query29	1244	554	551	551
query30	285	139	137	137
query31	853	702	698	698
query32	98	58	54	54
query33	711	236	227	227
query34	1055	488	496	488
query35	865	766	743	743
query36	931	805	887	805
query37	187	64	61	61
query38	3061	2958	3009	2958
query39	1305	1264	1250	1250
query40	277	104	102	102
query41	38	36	35	35
query42	109	97	95	95
query43	453	426	420	420
query44	1111	685	699	685
query45	189	181	172	172
query46	1052	785	788	785
query47	1568	1508	1442	1442
query48	413	353	350	350
query49	1184	295	299	295
query50	782	379	380	379
query51	4420	4269	4260	4260
query52	99	93	88	88
query53	334	270	270	270
query54	293	221	223	221
query55	83	79	76	76
query56	222	200	204	200
query57	969	889	867	867
query58	219	187	195	187
query59	2252	2111	1937	1937
query60	243	211	221	211
query61	86	89	85	85
query62	602	340	337	337
query63	280	253	262	253
query64	6579	2947	3135	2947
query65	3233	3219	3227	3219
query66	1423	315	319	315
query67	14423	14313	14285	14285
query68	4043	563	571	563
query69	510	378	369	369
query70	1265	1260	1220	1220
query71	322	251	252	251
query72	6035	2793	2656	2656
query73	736	325	330	325
query74	6284	6093	5907	5907
query75	3005	2431	2438	2431
query76	2553	1054	1137	1054
query77	328	232	234	232
query78	8939	8516	8424	8424
query79	1313	529	515	515
query80	1044	362	355	355
query81	466	197	198	197
query82	956	90	85	85
query83	216	122	116	116
query84	280	81	78	78
query85	1328	381	349	349
query86	392	304	298	298
query87	3294	3087	3105	3087
query88	2769	2389	2361	2361
query89	400	323	328	323
query90	1859	172	165	165
query91	148	125	124	124
query92	54	55	51	51
query93	1196	541	521	521
query94	1292	186	185	185
query95	446	348	346	346
query96	583	270	273	270
query97	4253	4099	4150	4099
query98	217	208	203	203
query99	1098	646	640	640
Total cold run time: 262245 ms
Total hot run time: 169099 ms

[doris-robot]

ClickBench: Total hot run time: 31.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a8d67e04fa3b8ea62a848304117eb294c55298c, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.05
query4	1.67	0.10	0.10
query5	0.52	0.51	0.52
query6	1.41	0.69	0.68
query7	0.01	0.01	0.02
query8	0.04	0.02	0.02
query9	0.56	0.50	0.50
query10	0.55	0.56	0.56
query11	0.13	0.10	0.10
query12	0.13	0.10	0.10
query13	0.63	0.61	0.61
query14	0.79	0.82	0.79
query15	0.83	0.81	0.82
query16	0.37	0.39	0.37
query17	0.99	0.99	1.00
query18	0.26	0.24	0.25
query19	1.78	1.71	1.73
query20	0.01	0.00	0.01
query21	15.40	0.67	0.58
query22	3.38	4.38	1.75
query23	17.48	1.10	1.02
query24	2.05	0.31	0.22
query25	0.41	0.07	0.19
query26	0.14	0.12	0.13
query27	0.06	0.04	0.04
query28	12.55	0.84	0.82
query29	12.66	3.25	3.19
query30	0.53	0.55	0.52
query31	2.77	0.34	0.35
query32	3.36	0.49	0.48
query33	3.21	3.15	3.26
query34	15.35	4.75	4.73
query35	4.73	4.74	4.75
query36	1.14	1.08	1.07
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.03	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 106.64 s
Total hot run time: 31.29 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0a8d67e04fa3b8ea62a848304117eb294c55298c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.6 seconds inserted 10000000 Rows, about 602K ops/s

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 35.69% (8548/23954)
Line Coverage: 27.48% (69372/252444)
Region Coverage: 26.64% (35978/135034)
Branch Coverage: 23.44% (18383/78420)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a8d67e04fa3b8ea62a848304117eb294c55298c_0a8d67e04fa3b8ea62a848304117eb294c55298c/report/index.html

[liaoxin01]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.