[Feature][Connector-V2] Add phoenix connector sink and phoenix connector source #2499

531651225 · 2022-08-22T10:34:55Z

#1946 Add phoenix sink connector
support thin driver client and Thick driver client

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

EricJoy2048 · 2022-08-22T10:58:09Z

Please update the pr title and add the [Connector-V2] label.

531651225 · 2022-08-22T11:00:23Z

Please update the pr title and add the [Connector-V2] label.

ok,done

.../java/org/apache/seatunnel/connectors/seatunnel/phoenix/client/PhoenixStatementExecutor.java

hailin0 · 2022-08-23T02:20:59Z

Can you add connector-v2 e2e-testcase?

reference：
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e

.../main/java/org/apache/seatunnel/connectors/seatunnel/phoenix/client/PhoenixOutputFormat.java

…seatunnel data to sink

531651225 · 2022-08-23T15:05:08Z

Can you add connector-v2 e2e-testcase?

reference： https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e

Ok, I will try to add it later. But the Phoenix Docker image is very large, I wonder if there is any problem

531651225 · 2022-08-25T02:07:33Z

Can you add connector-v2 e2e-testcase?

reference： https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e

E2E has been added and passed the test

...-v2-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/phoenix/FakeSourceToIoPhoenixIT.java

...l-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/phoenix/fakesource_to_phoenix.conf

...-v2-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/phoenix/FakeSourceToIoPhoenixIT.java

...l-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/phoenix/fakesource_to_phoenix.conf

...or-v2-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/phoenix/FakeSourceToPhoenixIT.java

...l-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/phoenix/fakesource_to_phoenix.conf

...-v2-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/phoenix/FakeSourceToIoPhoenixIT.java

hailin0 · 2022-08-25T02:53:53Z

Can phoenix drivers be loaded using jdbc connector?

531651225 · 2022-08-25T03:47:31Z

Can phoenix drivers be loaded using jdbc connector?

Can we still use this method ? Considering that more improvements will be added later, is this more flexible and less conflict

hailin0 · 2022-08-25T05:22:39Z

Can phoenix drivers be loaded using jdbc connector?

Can we still use this method ? Considering that more improvements will be added later, is this more flexible and less conflict

case #2429

531651225 · 2022-08-25T05:36:51Z

Can phoenix drivers be loaded using jdbc connector?

Can we still use this method ? Considering that more improvements will be added later, is this more flexible and less conflict

case #2429

Reuse JDBC connectors. Does the Phoenix connector need to be added to the connector JDBC module

hailin0 · 2022-08-25T06:32:17Z

Can phoenix drivers be loaded using jdbc connector?

Can we still use this method ? Considering that more improvements will be added later, is this more flexible and less conflict

case #2429

Reuse JDBC connectors. Does the Phoenix connector need to be added to the connector JDBC module

You need add database dialect to here

https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect

531651225 · 2022-08-26T23:26:27Z

Can phoenix drivers be loaded using jdbc connector?

Can we still use this method ? Considering that more improvements will be added later, is this more flexible and less conflict

case #2429

Reuse JDBC connectors. Does the Phoenix connector need to be added to the connector JDBC module

You need add database dialect to here

https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect

Thanks for your advice, I have re implemented the phoenix connector reuse jdbc connector in new commit. and and spark and flink e2e and pass the test

hailin0

Does Phoenix support XA transactions?
If unsupported then add note to documentation

...g/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/phoenix/PhoenixTypeMapper.java

...ache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/phoenix/PhoenixDialectFactory.java

...ink-connector-v2-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcPhoenixIT.java

531651225 · 2022-08-28T15:21:54Z

Does Phoenix support XA transactions? If unsupported then add note to documentation

i have added note to documentation in new commit

docs/en/connector-v2/sink/Phoenix.md

hailin0 · 2022-08-28T15:50:33Z

docs/en/connector-v2/source/Phoenix.md

+if you use  phoenix (thick) driver the value is  `org.apache.phoenix.jdbc.PhoenixDriver` or you use (thin) driver  the value is `org.apache.phoenix.queryserver.client.Driver`
+
+### url [string]
+if you use  phoenix (thick) driver the value is  `jdbc:phoenix:localhost:2182/hbase` or you use (thin) driver  the value is `jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF`


thanks，done

hailin0 · 2022-08-29T01:19:11Z

LGTM

531651225 · 2022-08-29T01:42:09Z

The CodeQL/Analyze (Java) step in CI was cancelled,. I think the overall compilation of the project timed out. How to solve it? @ashulin @hailin0

ashulin

LGTM.

* [Feature][Connector-V2] Add phoenix connector sink * fix doc style Co-authored-by: 毕博 <bibo@mafengwo.com>

毕博 added 3 commits August 22, 2022 18:26

[Feature][Connector-V2] Add phoenix connector sink

fde7911

Merge branch 'dev' into phoenix_connector-sink

33dc96e

[Feature][Connector-V2] Add pom.xml license

8d17b3a

531651225 changed the title ~~Phoenix connector sink~~ [Feature][Connector-V2] Add phoenix connector sink Aug 22, 2022

CalvinKirs added the connectors-v2 label Aug 22, 2022

ashulin reviewed Aug 22, 2022

View reviewed changes

.../java/org/apache/seatunnel/connectors/seatunnel/phoenix/client/PhoenixStatementExecutor.java Outdated Show resolved Hide resolved

hailin0 reviewed Aug 23, 2022

View reviewed changes

.../main/java/org/apache/seatunnel/connectors/seatunnel/phoenix/client/PhoenixOutputFormat.java Outdated Show resolved Hide resolved

[Feature][Connector-V2] base on phoenix supported datatype,transform …

a741e98

…seatunnel data to sink

[Feature][Connector-V2] phoenix connector add e2e

9c7181e

Merge branch 'dev' into phoenix_connector-sink

3d0266f

531651225 requested a review from hailin0 August 25, 2022 02:09

hailin0 reviewed Aug 25, 2022

View reviewed changes

fix checkstyle

1d44168

毕博 added 6 commits August 27, 2022 07:03

remove phoenix connector module and reuse jdbc connect

8e3a6a0

add e2e and passed

59f0987

add e2e and passed

0a01274

Merge branch 'dev' into phoenix_connector-sink

30347c8

add e2e and passed

2b5b9c0

add e2e and passed

04f1779

hailin0 reviewed Aug 28, 2022

View reviewed changes

...g/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/phoenix/PhoenixTypeMapper.java Outdated Show resolved Hide resolved

...ache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/phoenix/PhoenixDialectFactory.java Outdated Show resolved Hide resolved

hailin0 reviewed Aug 28, 2022

View reviewed changes

...ink-connector-v2-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcPhoenixIT.java Outdated Show resolved Hide resolved

fix code style and add improve testcase

187cda2

531651225 dismissed ashulin’s stale review via 187cda2 August 28, 2022 15:16

hailin0 reviewed Aug 28, 2022

View reviewed changes

fix doc style

f559c03

531651225 requested a review from ashulin August 29, 2022 01:39

ashulin approved these changes Aug 29, 2022

View reviewed changes

CalvinKirs merged commit 05ccf9d into apache:dev Aug 29, 2022

531651225 mentioned this pull request Aug 29, 2022

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Closed

531651225 changed the title ~~[Feature][Connector-V2] Add phoenix connector sink~~ [Feature][Connector-V2] Add phoenix connector sink and phoenix connector source Aug 29, 2022

iture123 mentioned this pull request Aug 29, 2022

[Feature][Connector-V2] new connecotor of Elasticsearch source(#2553) #2554

Closed

liugddx pushed a commit to liugddx/incubator-seatunnel that referenced this pull request Aug 29, 2022

[Feature][Connector-V2] Add phoenix connector sink (apache#2499)

d91433f

* [Feature][Connector-V2] Add phoenix connector sink * fix doc style Co-authored-by: 毕博 <bibo@mafengwo.com>

2013650523 mentioned this pull request Aug 30, 2022

[Connector-v2] add doris connector source #2536

Closed

3 tasks

Hisoka-X mentioned this pull request Aug 31, 2022

[Feature][Connector-V2][Translation] Support spark 3.3 #2574

Closed

3 tasks

wuchunfu mentioned this pull request Sep 2, 2022

[Feature][Connector-V2] Add mongodb connecter source #2596

Merged

3 tasks

531651225 mentioned this pull request Sep 3, 2022

[Feature][Connector-V2] add sqlserver connector #2622

Closed

3 tasks

This was referenced Sep 5, 2022

[Bug][flink-connector-elasticsearch7]sink failure caused by deprecated TransportClient #2543

Closed

[Feature][Connector-V2] Add sentry sink connector #2244 #2584

Merged

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 8, 2022

[Feature][Connector-V2] Add phoenix connector sink (apache#2499)

cb8cf66

* [Feature][Connector-V2] Add phoenix connector sink * fix doc style Co-authored-by: 毕博 <bibo@mafengwo.com>

531651225 deleted the phoenix_connector-sink branch September 9, 2022 03:22

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 15, 2022

[Feature][Connector-V2] Add phoenix connector sink (apache#2499)

45a73b8

* [Feature][Connector-V2] Add phoenix connector sink * fix doc style Co-authored-by: 毕博 <bibo@mafengwo.com>

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 16, 2022

[Feature][Connector-V2] Add phoenix connector sink (apache#2499)

bcac228

* [Feature][Connector-V2] Add phoenix connector sink * fix doc style Co-authored-by: 毕博 <bibo@mafengwo.com>

EricJoy2048 mentioned this pull request Oct 8, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

zhilinli123 mentioned this pull request Apr 11, 2023

[Feature][Connector-V2][Docs] Unified format for connector-v2 docs #4544

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] Add phoenix connector sink and phoenix connector source #2499

[Feature][Connector-V2] Add phoenix connector sink and phoenix connector source #2499

531651225 commented Aug 22, 2022

EricJoy2048 commented Aug 22, 2022

531651225 commented Aug 22, 2022

hailin0 commented Aug 23, 2022

531651225 commented Aug 23, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 26, 2022

hailin0 left a comment

531651225 commented Aug 28, 2022

hailin0 Aug 28, 2022

531651225 Aug 28, 2022

hailin0 commented Aug 29, 2022

531651225 commented Aug 29, 2022

ashulin left a comment

[Feature][Connector-V2] Add phoenix connector sink and phoenix connector source #2499

[Feature][Connector-V2] Add phoenix connector sink and phoenix connector source #2499

Conversation

531651225 commented Aug 22, 2022

Purpose of this pull request

Check list

EricJoy2048 commented Aug 22, 2022

531651225 commented Aug 22, 2022

hailin0 commented Aug 23, 2022

531651225 commented Aug 23, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 25, 2022

hailin0 commented Aug 25, 2022

531651225 commented Aug 26, 2022

hailin0 left a comment

Choose a reason for hiding this comment

531651225 commented Aug 28, 2022

hailin0 Aug 28, 2022

Choose a reason for hiding this comment

531651225 Aug 28, 2022

Choose a reason for hiding this comment

hailin0 commented Aug 29, 2022

531651225 commented Aug 29, 2022

ashulin left a comment

Choose a reason for hiding this comment