Skip to content

Conversation

@ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jul 23, 2024

Previously the azure sdk would be compiled into BE by default. But sometimes the BE would run in environment which only require AWS s3-compatible functionality. So user can use DISABLE_BUILD_AZURE to control whether build azure sdk into BE or not.

export DISABLE_BUILD_AZURE=ON to disable build with azure support

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

// specific language governing permissions and limitations
// under the License.

#include <fmt/core.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'fmt/core.h' file not found [clang-diagnostic-error]

#include <fmt/core.h>
         ^

@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1788:
build_azure() {
              ^-- SC1009 (info): The mentioned syntax error was in this brace group.


In thirdparty/build-thirdparty.sh line 1789:
    if [[ "${BUILD_AZURE}" =="OFF" ]]
    ^-- SC1049 (error): Did you forget the 'then' for this 'if'?
    ^-- SC1073 (error): Couldn't parse this if expression. Fix to allow more checks.
                             ^-- SC1035 (error): You are missing a required space here.


In thirdparty/build-thirdparty.sh line 1792:
    fi
    ^-- SC1050 (error): Expected 'then'.
      ^-- SC1072 (error): Unexpected keyword/token. Fix any mentioned problems and try again.

For more information:
  https://www.shellcheck.net/wiki/SC1035 -- You are missing a required space ...
  https://www.shellcheck.net/wiki/SC1049 -- Did you forget the 'then' for thi...
  https://www.shellcheck.net/wiki/SC1050 -- Expected 'then'.
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
thirdparty/build-thirdparty.sh:1789:28: test operator words must consist of a single literal
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1788:
build_azure() {
              ^-- SC1009 (info): The mentioned syntax error was in this brace group.


In thirdparty/build-thirdparty.sh line 1789:
    if [[ "${CBUILD_AZURE}" =="OFF" ]]
    ^-- SC1049 (error): Did you forget the 'then' for this 'if'?
    ^-- SC1073 (error): Couldn't parse this if expression. Fix to allow more checks.
                              ^-- SC1035 (error): You are missing a required space here.


In thirdparty/build-thirdparty.sh line 1792:
    fi
    ^-- SC1050 (error): Expected 'then'.
      ^-- SC1072 (error): Unexpected keyword/token. Fix any mentioned problems and try again.

For more information:
  https://www.shellcheck.net/wiki/SC1035 -- You are missing a required space ...
  https://www.shellcheck.net/wiki/SC1049 -- Did you forget the 'then' for thi...
  https://www.shellcheck.net/wiki/SC1050 -- Expected 'then'.
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
thirdparty/build-thirdparty.sh:1789:29: test operator words must consist of a single literal
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1788:
build_azure() {
              ^-- SC1009 (info): The mentioned syntax error was in this brace group.


In thirdparty/build-thirdparty.sh line 1789:
    if [[ "${CBUILD_AZURE}" =="OFF" ]]
    ^-- SC1049 (error): Did you forget the 'then' for this 'if'?
    ^-- SC1073 (error): Couldn't parse this if expression. Fix to allow more checks.
                              ^-- SC1035 (error): You are missing a required space here.


In thirdparty/build-thirdparty.sh line 1792:
    fi
    ^-- SC1050 (error): Expected 'then'.
      ^-- SC1072 (error): Unexpected keyword/token. Fix any mentioned problems and try again.

For more information:
  https://www.shellcheck.net/wiki/SC1035 -- You are missing a required space ...
  https://www.shellcheck.net/wiki/SC1049 -- Did you forget the 'then' for thi...
  https://www.shellcheck.net/wiki/SC1050 -- Expected 'then'.
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
thirdparty/build-thirdparty.sh:1789:29: test operator words must consist of a single literal
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1788:
build_azure() {
              ^-- SC1009 (info): The mentioned syntax error was in this brace group.


In thirdparty/build-thirdparty.sh line 1789:
    if [[ "${CBUILD_AZURE}" =="OFF" ]]
    ^-- SC1049 (error): Did you forget the 'then' for this 'if'?
    ^-- SC1073 (error): Couldn't parse this if expression. Fix to allow more checks.
                              ^-- SC1035 (error): You are missing a required space here.


In thirdparty/build-thirdparty.sh line 1792:
    else
    ^-- SC1050 (error): Expected 'then'.
        ^-- SC1072 (error): Unexpected keyword/token. Fix any mentioned problems and try again.

For more information:
  https://www.shellcheck.net/wiki/SC1035 -- You are missing a required space ...
  https://www.shellcheck.net/wiki/SC1049 -- Did you forget the 'then' for thi...
  https://www.shellcheck.net/wiki/SC1050 -- Expected 'then'.
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
thirdparty/build-thirdparty.sh:1789:29: test operator words must consist of a single literal
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40003 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db4b438b1f362e27760e7d85fb49fd4aa35f42d9, data reload: false

------ Round 1 ----------------------------------
q1	18083	4429	4355	4355
q2	2461	194	219	194
q3	12254	1171	1113	1113
q4	10252	740	820	740
q5	7547	2706	2646	2646
q6	222	145	145	145
q7	957	605	612	605
q8	9220	2080	2055	2055
q9	8813	6529	6629	6529
q10	8660	3779	3749	3749
q11	469	235	233	233
q12	391	230	219	219
q13	18770	2984	3009	2984
q14	273	246	241	241
q15	536	507	487	487
q16	520	393	377	377
q17	981	718	683	683
q18	8068	7544	7509	7509
q19	5833	1377	1398	1377
q20	715	337	313	313
q21	4944	3188	3166	3166
q22	341	283	291	283
Total cold run time: 120310 ms
Total hot run time: 40003 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4346	4233	4315	4233
q2	369	265	269	265
q3	2984	2734	2766	2734
q4	1861	1588	1648	1588
q5	5292	5287	5291	5287
q6	218	132	130	130
q7	2087	1744	1696	1696
q8	3206	3316	3288	3288
q9	8404	8371	8369	8369
q10	3950	3706	3694	3694
q11	579	467	493	467
q12	769	628	618	618
q13	16711	2954	3008	2954
q14	307	298	277	277
q15	522	477	483	477
q16	466	417	424	417
q17	1777	1473	1467	1467
q18	7546	7461	7349	7349
q19	1686	1562	1500	1500
q20	1999	1797	1771	1771
q21	4860	4671	4753	4671
q22	621	520	495	495
Total cold run time: 70560 ms
Total hot run time: 53747 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171959 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db4b438b1f362e27760e7d85fb49fd4aa35f42d9, data reload: false

query1	916	367	357	357
query2	6459	1852	1785	1785
query3	6652	206	220	206
query4	24749	17565	17660	17565
query5	4178	471	476	471
query6	257	174	159	159
query7	4599	290	287	287
query8	248	197	185	185
query9	8521	2459	2415	2415
query10	449	288	280	280
query11	12686	10045	10098	10045
query12	135	84	87	84
query13	1634	383	365	365
query14	10356	7379	6425	6425
query15	214	163	169	163
query16	7801	469	494	469
query17	1554	542	515	515
query18	1961	279	268	268
query19	198	151	157	151
query20	89	88	85	85
query21	208	127	149	127
query22	4503	4116	4069	4069
query23	34199	33510	33062	33062
query24	11839	2864	2826	2826
query25	649	363	364	363
query26	1770	173	147	147
query27	2965	271	281	271
query28	7689	2018	2002	2002
query29	1086	632	604	604
query30	286	144	147	144
query31	935	751	748	748
query32	93	52	55	52
query33	761	331	321	321
query34	915	486	493	486
query35	840	730	708	708
query36	1120	922	907	907
query37	167	75	79	75
query38	2842	2775	2787	2775
query39	854	802	799	799
query40	287	119	118	118
query41	46	45	46	45
query42	115	103	98	98
query43	499	454	452	452
query44	1214	726	721	721
query45	190	187	158	158
query46	1101	703	713	703
query47	1909	1783	1805	1783
query48	367	302	291	291
query49	1323	409	418	409
query50	791	400	387	387
query51	6754	6650	6615	6615
query52	102	95	87	87
query53	366	284	290	284
query54	855	442	439	439
query55	76	77	78	77
query56	285	259	279	259
query57	1190	1080	1038	1038
query58	258	240	261	240
query59	2840	2545	2504	2504
query60	296	305	287	287
query61	96	91	94	91
query62	813	662	677	662
query63	313	287	283	283
query64	10432	2185	1615	1615
query65	3171	3113	3126	3113
query66	1070	357	359	357
query67	15466	15058	14852	14852
query68	4828	555	547	547
query69	460	349	365	349
query70	1100	1156	1128	1128
query71	389	287	286	286
query72	7086	5935	5814	5814
query73	755	332	337	332
query74	6063	5671	5613	5613
query75	3590	2719	2706	2706
query76	2828	950	902	902
query77	562	312	309	309
query78	9660	9751	8973	8973
query79	2545	535	524	524
query80	1983	469	467	467
query81	564	230	232	230
query82	850	135	139	135
query83	273	167	169	167
query84	274	87	90	87
query85	2091	311	292	292
query86	329	319	325	319
query87	3296	3105	3137	3105
query88	3972	2420	2403	2403
query89	478	380	378	378
query90	1882	192	198	192
query91	132	99	101	99
query92	63	49	50	49
query93	3498	514	508	508
query94	831	380	286	286
query95	413	312	312	312
query96	608	280	275	275
query97	3201	3033	2999	2999
query98	212	199	197	197
query99	1592	1242	1270	1242
Total cold run time: 286010 ms
Total hot run time: 171959 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db4b438b1f362e27760e7d85fb49fd4aa35f42d9, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.07
query5	0.51	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.47
query10	0.54	0.55	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.58	0.57
query14	0.76	0.78	0.77
query15	0.84	0.80	0.81
query16	0.36	0.36	0.36
query17	0.97	1.01	1.01
query18	0.22	0.21	0.21
query19	1.83	1.81	1.70
query20	0.01	0.01	0.01
query21	15.40	0.77	0.65
query22	4.23	6.89	1.97
query23	18.24	1.49	1.20
query24	2.12	0.23	0.23
query25	0.16	0.08	0.08
query26	0.28	0.22	0.21
query27	0.45	0.24	0.23
query28	13.26	1.01	0.99
query29	12.61	3.30	3.30
query30	0.26	0.05	0.05
query31	2.87	0.38	0.38
query32	3.29	0.47	0.47
query33	2.87	2.96	2.97
query34	17.07	4.31	4.40
query35	4.40	4.41	4.42
query36	0.66	0.45	0.46
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.58 s

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 23, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions


#pragma once

#include <aws/core/client/AWSError.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'aws/core/client/AWSError.h' file not found [clang-diagnostic-error]

#include <aws/core/client/AWSError.h>
         ^

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26bbc02c8c1501864966a9806529486b0c4a2c5d, data reload: false

------ Round 1 ----------------------------------
q1	17660	4272	4274	4272
q2	2014	200	196	196
q3	10432	1179	1013	1013
q4	10199	761	816	761
q5	7605	2700	2648	2648
q6	226	141	137	137
q7	960	594	591	591
q8	9218	2057	2090	2057
q9	8739	6551	6547	6547
q10	8753	3753	3772	3753
q11	465	233	237	233
q12	527	217	216	216
q13	17786	2962	2982	2962
q14	287	228	251	228
q15	509	486	495	486
q16	500	393	389	389
q17	964	693	772	693
q18	8177	7592	7402	7402
q19	3599	1435	1399	1399
q20	677	327	323	323
q21	4939	3103	3346	3103
q22	352	289	287	287
Total cold run time: 114588 ms
Total hot run time: 39696 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4236	4179	4179
q2	363	255	269	255
q3	2991	2779	2879	2779
q4	1961	1711	1721	1711
q5	5600	5734	5550	5550
q6	220	131	135	131
q7	2196	1809	1829	1809
q8	3286	3420	3406	3406
q9	8755	8741	8837	8741
q10	4164	3850	3876	3850
q11	585	477	494	477
q12	783	600	608	600
q13	17088	3196	3187	3187
q14	322	272	299	272
q15	541	486	483	483
q16	476	443	434	434
q17	1803	1553	1502	1502
q18	8141	7819	7744	7744
q19	5832	1645	1529	1529
q20	2123	1927	1868	1868
q21	5090	4920	4798	4798
q22	605	490	496	490
Total cold run time: 77236 ms
Total hot run time: 55795 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174169 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26bbc02c8c1501864966a9806529486b0c4a2c5d, data reload: false

query1	909	371	368	368
query2	6432	1874	1819	1819
query3	6638	207	218	207
query4	28539	17511	17425	17425
query5	3639	482	460	460
query6	262	165	167	165
query7	4591	295	288	288
query8	251	207	200	200
query9	8497	2438	2427	2427
query10	446	270	293	270
query11	11412	10089	10006	10006
query12	115	87	82	82
query13	1637	383	356	356
query14	9463	7702	7697	7697
query15	223	165	166	165
query16	7610	475	491	475
query17	1593	556	519	519
query18	1713	275	265	265
query19	194	143	146	143
query20	90	88	79	79
query21	212	126	121	121
query22	4265	4178	3931	3931
query23	34112	33703	33507	33507
query24	10888	2906	2979	2906
query25	595	386	380	380
query26	696	159	149	149
query27	2205	274	275	274
query28	5869	2098	2108	2098
query29	905	653	623	623
query30	256	154	159	154
query31	975	740	777	740
query32	97	53	55	53
query33	724	342	339	339
query34	874	489	505	489
query35	860	777	770	770
query36	1137	983	974	974
query37	137	79	85	79
query38	3015	2884	2816	2816
query39	915	855	854	854
query40	200	120	115	115
query41	45	43	46	43
query42	110	95	97	95
query43	502	466	464	464
query44	1217	749	720	720
query45	197	160	158	158
query46	1089	730	732	730
query47	1888	1797	1817	1797
query48	360	289	285	285
query49	824	403	403	403
query50	772	387	388	387
query51	6767	6651	6629	6629
query52	99	93	86	86
query53	363	286	281	281
query54	894	449	443	443
query55	73	72	72	72
query56	291	270	264	264
query57	1160	1064	1072	1064
query58	240	245	262	245
query59	2878	2527	2468	2468
query60	293	287	273	273
query61	98	90	139	90
query62	818	639	625	625
query63	316	290	287	287
query64	9094	2207	1632	1632
query65	3146	3086	3121	3086
query66	753	322	342	322
query67	15307	14952	14971	14952
query68	5160	545	546	545
query69	634	434	364	364
query70	1179	1138	1167	1138
query71	470	281	271	271
query72	7292	5512	5835	5512
query73	772	327	325	325
query74	6276	5685	5615	5615
query75	3673	2670	2716	2670
query76	3616	998	1011	998
query77	613	324	302	302
query78	9645	10505	9598	9598
query79	2308	532	510	510
query80	1048	478	464	464
query81	580	222	217	217
query82	509	132	129	129
query83	325	162	160	160
query84	279	93	86	86
query85	725	306	294	294
query86	471	309	311	309
query87	3379	3089	3122	3089
query88	3561	2366	2379	2366
query89	467	396	369	369
query90	1768	195	191	191
query91	130	103	115	103
query92	57	50	48	48
query93	1899	514	514	514
query94	992	293	293	293
query95	402	312	315	312
query96	605	274	283	274
query97	3263	2979	3013	2979
query98	225	193	192	192
query99	1593	1307	1243	1243
Total cold run time: 276701 ms
Total hot run time: 174169 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26bbc02c8c1501864966a9806529486b0c4a2c5d, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.07
query5	0.49	0.47	0.48
query6	1.14	0.73	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.50
query10	0.55	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.75	0.79	0.78
query15	0.84	0.80	0.82
query16	0.37	0.37	0.36
query17	1.01	1.04	1.03
query18	0.23	0.22	0.21
query19	1.83	1.73	1.78
query20	0.01	0.01	0.01
query21	15.39	0.76	0.64
query22	4.15	7.92	1.99
query23	18.28	1.40	1.26
query24	2.15	0.23	0.23
query25	0.15	0.08	0.09
query26	0.30	0.22	0.21
query27	0.45	0.23	0.22
query28	13.24	1.00	1.00
query29	12.62	3.31	3.29
query30	0.25	0.05	0.06
query31	3.14	0.38	0.39
query32	3.26	0.48	0.46
query33	2.87	2.93	2.86
query34	17.03	4.30	4.37
query35	4.38	4.45	4.43
query36	0.66	0.48	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.8 s
Total hot run time: 30.64 s

Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. tested on arm fine.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@ByteYue ByteYue requested a review from gavinchou July 23, 2024 15:28
@zclllyybb
Copy link
Contributor

run arm

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit c14320c into apache:master Jul 25, 2024
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
Previously the azure sdk would be compiled into BE by default. But
sometimes the BE would run in environment which only require AWS
s3-compatible functionality. So user can use `DISABLE_BUILD_AZURE` to
control whether build azure sdk into BE or not.

`export DISABLE_BUILD_AZURE=ON` to disable build with azure support
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants