[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug #27661

xinyiZzz · 2023-11-27T15:49:53Z

Proposed changes

Design Documentation Linked to #25514

Support jdbc:arrow-flight-sql, which allows Java to connect to Doris Arrow Flight SQL Server using JDBC protocol.. see: https://www.dremio.com/drivers/jdbc/, https://arrow.apache.org/adbc/main/driver/jdbc.html, https://arrow.apache.org/cookbook/java/jdbc.html, https://arrow.apache.org/docs/java/flight_sql.html
Supports PreparedStatement, but can only execute complete SQL, not support SQL parameters. Because JDBC can only use PreparedStatement to execute query, see: [Java] jdbc:arrow-flight-sql how to execute query without using PreparedStatement arrow#38742
Temporarily upgrade Arrow to dev version 15.0.0-SNAPSHOT, because the latest release version Arrow 14.0.1 jdbc:arrow-flight-sql has BUG, see: [Java] jdbc:arrow-flight-sql executeQuery behaves incorrectly arrow#38785
Support query results with anonymous columns
Fix ResultSinkType error
Support explain stmt
Supports separately specifying BE’s Arrow Flight Server Host
...

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-11-27T15:57:14Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-11-27T17:18:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-11-27T18:22:06Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-11-27T18:44:34Z

run buildall

github-actions · 2023-11-27T18:51:58Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-11-27T20:41:10Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 2986657a4b5878ee563c5cbb730aabe732d0f771, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4887	4653	4648	4648
q2	371	144	161	144
q3	2067	1896	1883	1883
q4	1385	1247	1243	1243
q5	3941	3937	3969	3937
q6	259	132	132	132
q7	1393	879	883	879
q8	2796	2808	2775	2775
q9	10207	10259	10288	10259
q10	3441	3540	3529	3529
q11	371	254	254	254
q12	442	298	304	298
q13	4564	3826	3836	3826
q14	324	284	284	284
q15	578	538	515	515
q16	662	579	579	579
q17	1148	974	935	935
q18	7910	7566	7420	7420
q19	1686	1695	1671	1671
q20	573	308	314	308
q21	4446	4029	3967	3967
q22	468	373	379	373
Total cold run time: 53919 ms
Total hot run time: 49859 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4595	4553	4566	4553
q2	357	240	269	240
q3	4046	4017	4014	4014
q4	2733	2716	2709	2709
q5	9713	9645	9677	9645
q6	242	124	125	124
q7	3051	2498	2517	2498
q8	4475	4445	4450	4445
q9	13304	13185	13148	13148
q10	4035	4169	4156	4156
q11	783	655	640	640
q12	993	801	815	801
q13	4289	3557	3614	3557
q14	370	345	343	343
q15	571	521	516	516
q16	745	672	698	672
q17	3880	3870	3871	3870
q18	9637	9138	9126	9126
q19	1795	1765	1771	1765
q20	2414	2038	2038	2038
q21	8762	8646	8768	8646
q22	872	789	830	789
Total cold run time: 81662 ms
Total hot run time: 78295 ms

doris-robot · 2023-11-27T20:52:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.54 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 28 seconds loaded 2358488459 Bytes, about 80 MB/s
stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098784136 Bytes

be/src/util/arrow/row_batch.cpp

be/src/common/config.cpp

fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java

gensrc/proto/internal_service.proto

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java

github-actions · 2023-11-29T03:49:52Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-11-29T04:01:31Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-11-29T04:33:56Z

run buildall

xinyiZzz · 2023-11-29T04:38:46Z

run buildall

github-actions · 2023-11-29T04:41:31Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-11-29T04:45:40Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions · 2023-11-29T05:00:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-11-29T05:00:39Z

PR approved by anyone and no changes requested.

github-actions · 2023-11-29T05:02:58Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-11-29T05:41:25Z

run buildall

github-actions · 2023-11-29T05:48:55Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-11-29T07:50:26Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.75 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17100899685 Bytes

wangbo

LGTM

… and Fix Bug (apache#27661)

1

69db432

2

161bfad

github-actions bot added the meta-change label Nov 27, 2023

3

45205c6

4

2986657

yiguolei reviewed Nov 29, 2023

View reviewed changes

be/src/util/arrow/row_batch.cpp Show resolved Hide resolved

yiguolei reviewed Nov 29, 2023

View reviewed changes

be/src/common/config.cpp Outdated Show resolved Hide resolved

yiguolei reviewed Nov 29, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java Show resolved Hide resolved

yiguolei reviewed Nov 29, 2023

View reviewed changes

gensrc/proto/internal_service.proto Outdated Show resolved Hide resolved

yiguolei reviewed Nov 29, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated Show resolved Hide resolved

yiguolei reviewed Nov 29, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java Show resolved Hide resolved

5

287056d

xinyiZzz force-pushed the 20231110_flight_add_regression_test branch from 6b37548 to 287056d Compare November 29, 2023 03:53

6

2b0a283

7

4b1adf9

yiguolei approved these changes Nov 29, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 29, 2023

github-actions bot added the reviewed label Nov 29, 2023

Merge branch 'master' into 20231110_flight_add_regression_test

6cdbb4b

wm1581066 requested a review from eldenmoon November 29, 2023 11:44

wangbo approved these changes Nov 29, 2023

View reviewed changes

yiguolei merged commit d96e2df into apache:master Nov 29, 2023
25 checks passed

xinyiZzz mentioned this pull request Dec 1, 2023

[Feature] Doris support Arrow Flight SQL protocol #25514

Open

1 task

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement…

4e4ff8a

… and Fix Bug (apache#27661)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug #27661

[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug #27661

xinyiZzz commented Nov 27, 2023 •

edited

github-actions bot commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

xinyiZzz commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

doris-robot commented Nov 27, 2023

doris-robot commented Nov 27, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

yiguolei left a comment

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

doris-robot commented Nov 29, 2023

wangbo left a comment

[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug #27661

[feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug #27661

Conversation

xinyiZzz commented Nov 27, 2023 • edited

Proposed changes

Further comments

github-actions bot commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

xinyiZzz commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

doris-robot commented Nov 27, 2023

doris-robot commented Nov 27, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

xinyiZzz commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

doris-robot commented Nov 29, 2023

wangbo left a comment

Choose a reason for hiding this comment

xinyiZzz commented Nov 27, 2023 •

edited