issues#1946 add [Druid] Source & Sink #2544

guanboo · 2022-08-27T06:59:22Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

guanboo · 2022-08-27T07:03:01Z

fix:issues#1946 add [Druid] Source & Sink

guanboo · 2022-08-27T07:03:29Z

fix:issues#1946 add [Druid] Source & Sink

ashulin

Please add the license header.
Maybe you can add druid dialect in jdbc connector.

ashulin · 2022-08-27T15:37:26Z

seatunnel-connectors-v2/connector-druid/pom.xml

@@ -0,0 +1,58 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"


add license header

ashulin · 2022-08-27T15:37:42Z

...d/src/main/java/org/apache/seatunnel/connectors/seatunnel/druid/client/DruidInputFormat.java

@@ -0,0 +1,201 @@
+package org.apache.seatunnel.connectors.seatunnel.druid.client;


ashulin · 2022-08-27T15:44:44Z

@ic4y PTAL, thx.

CalvinKirs · 2022-08-28T11:21:12Z

seatunnel-connectors-v2/connector-druid/pom.xml

+        <dependency>
+            <groupId>org.apache.druid</groupId>
+            <artifactId>druid-indexing-service</artifactId>
+            <scope>compile</scope>


We already have the class JsonUtil, please use this.

In JsonUtil OBJECT_MAPPER configure state cannot be shared.need new another ObjectMapper

CalvinKirs · 2022-08-28T11:22:17Z

seatunnel-connectors-v2/connector-druid/pom.xml

+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>connector-common</artifactId>
+            <version>${project.version}</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+            <version>4.5.13</version>
+            <scope>compile</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.calcite.avatica</groupId>
+            <artifactId>avatica-core</artifactId>
+            <version>1.15.0</version>
+            <scope>compile</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.druid</groupId>
+            <artifactId>druid-core</artifactId>
+            <version>RELEASE</version>
+            <scope>compile</scope>


The dependency version should manage by root pom

CalvinKirs · 2022-08-28T11:23:57Z

...d/src/main/java/org/apache/seatunnel/connectors/seatunnel/druid/client/DruidInputFormat.java

+        } catch (Exception e) {
+            LOGGER.warn("get row type info exception", e);


Should throw exception?

hailin0

Can you add connector-v2 e2e-testcase & docs

reference：#2499

ic4y · 2022-08-29T09:30:00Z

...d/src/main/java/org/apache/seatunnel/connectors/seatunnel/druid/client/DruidInputFormat.java

+/**
+ * guanbo
+ */


delete meaningless comments

ic4y · 2022-08-29T09:37:34Z

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/druid/client/DruidOutputFormat.java

+/**
+ * guanbo
+ */


ic4y · 2022-08-29T09:52:04Z

...id/src/main/java/org/apache/seatunnel/connectors/seatunnel/druid/config/DruidSinkConfig.java

+/**
+ * guanbo
+ */


Check all guanbo places, delete it.

…#1946_druid

…ions columns type

…mat columns type

CalvinKirs · 2022-09-02T03:11:07Z

Hi, please resolve conflicts files

…cts files

guanboo · 2022-09-02T05:58:35Z

done
resolve conflicts files：
seatunnel-connectors-v2/pom.xml
pom.xml
plugin-mapping.properties

…k doc and DEFAULT_TIMESTAMP_MISSING_VALUE assign initial value

guanboo · 2022-09-04T15:29:38Z

done add source & sink doc

Hisoka-X · 2022-09-05T06:10:57Z

config/spark.streaming.conf.template

@@ -26,7 +26,7 @@ env {
  spark.executor.instances = 2
  spark.executor.cores = 1
  spark.executor.memory = "1g"
-  spark.stream.batchDuration = 5
+  spark.streaming.batchDuration = 5


Please check this #2575

Hisoka-X · 2022-09-05T06:12:57Z

docs/en/connector-v2/sink/Druid.md

@@ -0,0 +1,101 @@
+# Druid


Please update doc to match new format, reference: #2625

Hisoka-X · 2022-09-05T06:13:38Z

docs/en/contribution/setup.md

@@ -35,12 +35,6 @@ Otherwise, your code could not start in JetBrains IntelliJ IDEA correctly.
 ./mvnw install -Dmaven.test.skip
 ```

-### Building SeaTunnel from source


Why remove this?

Hisoka-X · 2022-09-05T06:13:57Z

docs/zh-CN/spark/quick-start.md

@@ -34,7 +34,7 @@ ln -s seatunnel-<version> seatunnel
 ```bash
 env {
  # seatunnel defined streaming batch duration in seconds
-  spark.stream.batchDuration = 5


same as above

Hisoka-X · 2022-09-05T06:22:19Z

pom.xml

@@ -204,6 +204,11 @@
        <p3c-pmd.version>1.3.0</p3c-pmd.version>
        <maven-scm-provider-jgit.version>1.9.5</maven-scm-provider-jgit.version>
        <testcontainer.version>1.16.3</testcontainer.version>
+        <jackson-datatype-joda.version>2.12.1</jackson-datatype-joda.version>


Don't add the connector's dependency in root pom, just use it in connector's pom.xml, reference: https://github.com/apache/incubator-seatunnel/pull/2630/files

Hisoka-X · 2022-09-05T06:23:55Z

...nel-api-spark/src/main/scala/org/apache/seatunnel/spark/stream/SparkStreamingExecution.scala

@@ -48,12 +48,9 @@ class SparkStreamingExecution(sparkEnvironment: SparkEnvironment)
            dataset)
        }
        var ds = dataset
-
-        if (ds.take(1).length > 0) {


Can't delete this, check this: #2578

Hisoka-X · 2022-09-05T06:28:34Z

I'm sorry, but I want reject this PR. Because too many code update but no connection with Druid Source/Sink. You can split it into multi PR, each have only one main purpose. Thanks again!

Hisoka-X · 2022-09-05T06:30:07Z

And you delete lots of code which shouldn't be deleted.

Hisoka-X · 2022-09-05T06:31:42Z

Any update you can reopen this PR again, or create another PR only with Druid Source/Sink (recommend).

guanboo · 2022-09-05T06:36:39Z

OK THX！
I can create another PR only with Druid Source/Sink.Done today or tomorrow

guanboo · 2022-09-05T12:57:17Z

done
#2651

fix:add [Druid] Source & Sink,issues#1946

3e0a238

guanboo closed this Aug 27, 2022

guanboo reopened this Aug 27, 2022

guanboo mentioned this pull request Aug 27, 2022

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Closed

ashulin reviewed Aug 27, 2022

View reviewed changes

CalvinKirs reviewed Aug 28, 2022

View reviewed changes

hailin0 reviewed Aug 29, 2022

View reviewed changes

Merge branch 'dev' into Feature#1946_druid

87807d1

ic4y reviewed Aug 29, 2022

View reviewed changes

bo.guan added 5 commits August 30, 2022 17:07

FIX: Modify the issues#1946 add [Druid] Source & Sink #2544

cb6828b

Merge remote-tracking branch 'origin/Feature#1946_druid' into Feature…

6a90e5c

…#1946_druid

fix:Modify issues#1946 add [Druid] Source & Sink #2544

de346f5

fix:Modify issues#1946 add [Druid] Source & Sink #2544 DruidSourceOpt…

69d9436

…ions columns type

fix:Modify issues#1946 add [Druid] Source & Sink #2544 DruidOutputFor…

4f2bc23

…mat columns type

fix:Modify issues#1946 add [Druid] Source & Sink #2544 resolve confli…

17e5c9a

…cts files

fix:Modify issues#1946 add [Druid] Source & Sink #2544 add source/sin…

756e14f

…k doc and DEFAULT_TIMESTAMP_MISSING_VALUE assign initial value

Hisoka-X reviewed Sep 5, 2022

View reviewed changes

Hisoka-X closed this Sep 5, 2022

Hisoka-X added the invalid label Sep 5, 2022

		@@ -0,0 +1,58 @@
		<?xml version="1.0" encoding="UTF-8"?>
		<project xmlns="http://maven.apache.org/POM/4.0.0"

		@@ -0,0 +1,201 @@
		package org.apache.seatunnel.connectors.seatunnel.druid.client;

		} catch (Exception e) {
		LOGGER.warn("get row type info exception", e);

issues#1946 add [Druid] Source & Sink #2544

issues#1946 add [Druid] Source & Sink #2544

Conversation

guanboo commented Aug 27, 2022

Purpose of this pull request

Check list

guanboo commented Aug 27, 2022

guanboo commented Aug 27, 2022

ashulin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashulin commented Aug 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CalvinKirs commented Sep 2, 2022

guanboo commented Sep 2, 2022

guanboo commented Sep 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Sep 5, 2022

Hisoka-X commented Sep 5, 2022

Hisoka-X commented Sep 5, 2022

guanboo commented Sep 5, 2022

guanboo commented Sep 5, 2022

ashulin left a comment •

edited

Loading