Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](statistics)Use utf-8 charset for internal query result. #39989

Merged
merged 1 commit into from
Aug 28, 2024
Merged

[fix](statistics)Use utf-8 charset for internal query result. #39989

merged 1 commit into from
Aug 28, 2024

Conversation

Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Aug 27, 2024 •
edited
Loading

Use utf-8 charset for internal query result.
User's environment may set default java file encode to some other character set, which may cause internal query get Garbled Characters. Use utf-8 to decode string bytes from BE.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review August 27, 2024 09:15
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38265 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b4a73c49ce5544992211e8b09221a5902fe9109, data reload: false

------ Round 1 ----------------------------------
q1	17821	5426	4346	4346
q2	2645	184	177	177
q3	11521	1176	1099	1099
q4	10563	790	790	790
q5	8072	2890	2849	2849
q6	234	140	146	140
q7	967	631	619	619
q8	9638	2070	2036	2036
q9	7271	6546	6555	6546
q10	7004	2174	2221	2174
q11	457	245	247	245
q12	394	227	223	223
q13	17762	3052	3008	3008
q14	273	240	231	231
q15	530	482	495	482
q16	485	396	396	396
q17	958	668	687	668
q18	7447	6875	6952	6875
q19	1394	1077	973	973
q20	678	325	344	325
q21	3945	3028	3099	3028
q22	1125	1035	1048	1035
Total cold run time: 111184 ms
Total hot run time: 38265 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4326	4255	4255
q2	395	291	275	275
q3	2914	2713	2657	2657
q4	1945	1692	1681	1681
q5	5427	5389	5382	5382
q6	218	130	132	130
q7	2093	1761	1752	1752
q8	3196	3365	3348	3348
q9	8418	8475	8387	8387
q10	3488	3233	3188	3188
q11	596	506	523	506
q12	771	613	618	613
q13	11425	3070	3037	3037
q14	312	275	283	275
q15	527	507	476	476
q16	491	422	422	422
q17	1784	1503	1459	1459
q18	7799	7465	7359	7359
q19	1658	1513	1480	1480
q20	2042	1833	1826	1826
q21	5510	5201	5206	5201
q22	1154	1061	1041	1041
Total cold run time: 66517 ms
Total hot run time: 54750 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188005 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b4a73c49ce5544992211e8b09221a5902fe9109, data reload: false

query1	906	377	367	367
query2	6458	1982	1910	1910
query3	6650	214	221	214
query4	33874	23367	23203	23203
query5	4160	499	479	479
query6	254	165	169	165
query7	4598	292	291	291
query8	266	214	198	198
query9	8498	2473	2483	2473
query10	425	260	264	260
query11	16390	14986	15064	14986
query12	157	99	100	99
query13	1650	397	376	376
query14	9831	7186	6773	6773
query15	283	176	177	176
query16	8027	470	493	470
query17	1573	560	543	543
query18	2074	298	304	298
query19	271	142	142	142
query20	113	110	116	110
query21	204	102	101	101
query22	4392	4192	4301	4192
query23	33969	33567	33260	33260
query24	11144	2933	2871	2871
query25	619	383	388	383
query26	1144	164	162	162
query27	2457	290	277	277
query28	7406	2049	2033	2033
query29	804	420	409	409
query30	304	162	155	155
query31	940	735	815	735
query32	100	59	59	59
query33	785	302	313	302
query34	939	478	485	478
query35	904	730	700	700
query36	1089	957	929	929
query37	160	85	84	84
query38	4012	3819	3828	3819
query39	1435	1379	1415	1379
query40	205	118	117	117
query41	49	47	45	45
query42	115	106	96	96
query43	510	477	460	460
query44	1214	781	750	750
query45	196	163	169	163
query46	1109	755	755	755
query47	1896	1764	1792	1764
query48	363	295	296	295
query49	1092	427	448	427
query50	813	415	423	415
query51	7163	7094	6984	6984
query52	98	91	91	91
query53	255	186	180	180
query54	893	465	447	447
query55	80	75	74	74
query56	276	256	248	248
query57	1196	1075	1069	1069
query58	233	228	239	228
query59	3011	2895	2953	2895
query60	297	266	263	263
query61	104	99	101	99
query62	838	667	667	667
query63	217	186	186	186
query64	4203	668	679	668
query65	3250	3139	3180	3139
query66	1288	357	331	331
query67	15715	15262	15395	15262
query68	3382	610	602	602
query69	421	281	284	281
query70	1140	1155	1115	1115
query71	344	287	279	279
query72	6521	4242	4173	4173
query73	750	343	337	337
query74	9038	8887	9105	8887
query75	3411	2740	2712	2712
query76	1906	949	1072	949
query77	516	316	322	316
query78	9537	9008	9049	9008
query79	1044	550	553	550
query80	707	545	521	521
query81	507	235	228	228
query82	239	145	140	140
query83	180	158	176	158
query84	235	91	87	87
query85	775	349	394	349
query86	313	343	278	278
query87	4359	4233	4186	4186
query88	2912	2384	2364	2364
query89	375	290	281	281
query90	1970	216	207	207
query91	128	102	106	102
query92	63	52	52	52
query93	1035	538	541	538
query94	861	302	269	269
query95	355	262	256	256
query96	592	267	274	267
query97	3207	3072	3080	3072
query98	216	207	203	203
query99	1460	1259	1281	1259
Total cold run time: 284443 ms
Total hot run time: 188005 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b4a73c49ce5544992211e8b09221a5902fe9109, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.51	0.48	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.56	0.55	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.61	0.59	0.58
query14	0.76	0.80	0.80
query15	0.90	0.81	0.82
query16	0.36	0.37	0.38
query17	1.06	1.04	0.99
query18	0.21	0.20	0.21
query19	1.90	1.86	1.76
query20	0.01	0.01	0.01
query21	15.40	0.66	0.64
query22	4.23	6.20	2.22
query23	18.30	1.29	1.28
query24	2.10	0.23	0.22
query25	0.15	0.08	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.24	1.03	1.01
query29	12.65	3.35	3.31
query30	0.24	0.07	0.06
query31	2.85	0.40	0.39
query32	3.27	0.48	0.48
query33	2.95	3.01	2.98
query34	17.09	4.46	4.48
query35	4.50	4.44	4.42
query36	0.66	0.46	0.49
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.69 s
Total hot run time: 31.12 s

@Jibing-Li Jibing-Li changed the title [fix](fe)Add java option to use utf8. [fix](statistics)Use utf-8 charset for internal query result. Aug 27, 2024
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37667 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bffce6d6d5354d365cae6453ce37a7c31cb3ca03, data reload: false

------ Round 1 ----------------------------------
q1	17615	4381	4287	4287
q2	2024	182	178	178
q3	11695	954	1160	954
q4	10518	767	788	767
q5	7750	2843	2800	2800
q6	227	138	138	138
q7	970	619	598	598
q8	9353	2060	2053	2053
q9	7315	6473	6557	6473
q10	6994	2256	2179	2179
q11	477	250	244	244
q12	393	222	218	218
q13	18844	3061	3035	3035
q14	295	243	233	233
q15	518	480	486	480
q16	492	395	399	395
q17	973	663	724	663
q18	7246	6800	6877	6800
q19	1383	1017	1094	1017
q20	709	332	329	329
q21	3842	3091	2839	2839
q22	1077	1038	987	987
Total cold run time: 110710 ms
Total hot run time: 37667 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4309	4304	4222	4222
q2	373	275	279	275
q3	2881	2611	2628	2611
q4	1910	1636	1605	1605
q5	5592	5652	5727	5652
q6	234	135	134	134
q7	2255	1888	1818	1818
q8	3328	3430	3493	3430
q9	8830	8863	8760	8760
q10	3635	3399	3318	3318
q11	628	533	522	522
q12	836	659	652	652
q13	15649	3220	3228	3220
q14	331	283	288	283
q15	552	479	501	479
q16	533	457	457	457
q17	1814	1523	1516	1516
q18	8012	7728	7755	7728
q19	1705	1506	1516	1506
q20	2147	1956	1865	1865
q21	5685	5545	5461	5461
q22	1140	1031	1048	1031
Total cold run time: 72379 ms
Total hot run time: 56545 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193858 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bffce6d6d5354d365cae6453ce37a7c31cb3ca03, data reload: false

query1	1288	899	891	891
query2	6374	1989	1962	1962
query3	10626	4177	3826	3826
query4	59682	26876	23302	23302
query5	5406	504	524	504
query6	406	164	165	164
query7	5775	304	303	303
query8	299	225	214	214
query9	9005	2487	2515	2487
query10	501	283	268	268
query11	15977	15311	15498	15311
query12	169	108	101	101
query13	1562	406	383	383
query14	11043	6836	7583	6836
query15	263	176	181	176
query16	7551	460	533	460
query17	1164	598	600	598
query18	1800	326	324	324
query19	308	170	157	157
query20	122	117	122	117
query21	212	107	110	107
query22	4717	4412	4590	4412
query23	34426	34044	33616	33616
query24	5999	2911	2927	2911
query25	561	409	414	409
query26	704	167	162	162
query27	1767	298	286	286
query28	3796	2067	2045	2045
query29	695	432	436	432
query30	245	167	171	167
query31	959	727	752	727
query32	86	56	57	56
query33	495	299	299	299
query34	890	506	511	506
query35	865	749	712	712
query36	1080	948	962	948
query37	141	88	86	86
query38	4080	3828	3911	3828
query39	1443	1408	1454	1408
query40	206	123	123	123
query41	48	51	45	45
query42	113	100	96	96
query43	512	476	480	476
query44	1123	757	769	757
query45	199	169	171	169
query46	1119	740	787	740
query47	1896	1771	1806	1771
query48	378	316	309	309
query49	781	440	456	440
query50	808	424	435	424
query51	7319	7109	7181	7109
query52	99	92	90	90
query53	256	188	187	187
query54	582	473	481	473
query55	81	79	84	79
query56	287	270	266	266
query57	1217	1083	1100	1083
query58	233	243	256	243
query59	2951	2781	2661	2661
query60	310	282	287	282
query61	123	121	123	121
query62	781	629	667	629
query63	211	188	192	188
query64	2952	798	680	680
query65	3340	3183	3227	3183
query66	680	336	352	336
query67	15248	15405	15224	15224
query68	4431	561	565	561
query69	431	277	279	277
query70	1207	1131	1060	1060
query71	375	292	278	278
query72	6658	4039	4033	4033
query73	747	340	341	340
query74	9241	8953	8897	8897
query75	3448	2723	2735	2723
query76	1804	1046	982	982
query77	562	325	334	325
query78	10188	9331	9532	9331
query79	1077	559	543	543
query80	829	520	517	517
query81	551	240	237	237
query82	292	138	138	138
query83	220	152	151	151
query84	259	78	83	78
query85	835	295	303	295
query86	326	286	300	286
query87	4505	4394	4205	4205
query88	2998	2411	2389	2389
query89	402	292	283	283
query90	1826	197	198	197
query91	130	109	99	99
query92	64	52	54	52
query93	1117	542	540	540
query94	750	301	290	290
query95	362	260	264	260
query96	593	278	273	273
query97	3298	3113	3081	3081
query98	223	212	204	204
query99	1745	1248	1256	1248
Total cold run time: 307829 ms
Total hot run time: 193858 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bffce6d6d5354d365cae6453ce37a7c31cb3ca03, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.52	0.50	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.55
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.62	0.58	0.59
query14	0.77	0.82	0.78
query15	0.84	0.82	0.82
query16	0.35	0.39	0.37
query17	1.07	1.07	1.06
query18	0.21	0.20	0.20
query19	1.96	1.86	1.81
query20	0.02	0.01	0.01
query21	15.39	0.66	0.65
query22	4.50	6.95	2.14
query23	18.30	1.40	1.20
query24	2.06	0.20	0.20
query25	0.15	0.08	0.08
query26	0.25	0.18	0.18
query27	0.08	0.08	0.07
query28	13.30	1.03	0.99
query29	12.66	3.25	3.27
query30	0.25	0.06	0.05
query31	2.88	0.40	0.38
query32	3.27	0.46	0.47
query33	2.97	2.96	3.00
query34	17.13	4.38	4.39
query35	4.41	4.47	4.50
query36	0.66	0.46	0.49
query37	0.18	0.16	0.15
query38	0.14	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.97 s
Total hot run time: 30.87 s

@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li
Copy link
Contributor Author

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly englefly merged commit 1a3db44 into apache:master Aug 28, 2024
30 of 32 checks passed
@Jibing-Li Jibing-Li deleted the utf8 branch August 28, 2024 05:29
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 28, 2024
@Jibing-Li
[fix](statistics)Use utf-8 charset for internal query result. (apache…
cd3d980 
…#39989)

Use utf-8 charset for internal query result.
User's environment may set default java file encode to some other
character set, which may cause internal query get Garbled Characters.
Use utf-8 to decode string bytes from BE.
@Jibing-Li Jibing-Li mentioned this pull request Aug 28, 2024
Merged
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 28, 2024
@Jibing-Li
[fix](statistics)Use utf-8 charset for internal query result. (apache…
7ed9ac5 
…#39989)

Use utf-8 charset for internal query result.
User's environment may set default java file encode to some other
character set, which may cause internal query get Garbled Characters.
Use utf-8 to decode string bytes from BE.
@Jibing-Li Jibing-Li mentioned this pull request Aug 28, 2024
Merged
Jibing-Li added a commit that referenced this pull request Aug 28, 2024
@Jibing-Li
[fix](statistics)Use utf-8 charset for internal query result. (#39989) (
ad998ef 
#40025)

backport: #39989
Jibing-Li added a commit that referenced this pull request Aug 28, 2024
@Jibing-Li
[fix](statistics)Use utf-8 charset for internal query result. (#39989) (
b3b085c 
#40026)

backport: #39989
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@Jibing-Li @dataroaring
[fix](statistics)Use utf-8 charset for internal query result. (#39989)
4a6e4dd 
Use utf-8 charset for internal query result.
User's environment may set default java file encode to some other
character set, which may cause internal query get Garbled Characters.
Use utf-8 to decode string bytes from BE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@englefly englefly englefly approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Jibing-Li @doris-robot @englefly @morrySnow