[Feature][Connector-V2][Translation] Support spark 3.3 #2574

zhaomin1423 · 2022-08-30T18:05:14Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

ashulin

Please add licenses according to Dependency licenses action.
you can see https://seatunnel.apache.org/docs/contribution/new-license

Hisoka-X · 2022-08-31T04:10:35Z

Can you add e2e from spark3 and spark2?
reference：#2499

zhaomin1423 · 2022-09-05T16:20:57Z

Can you add e2e from spark3 and spark2? reference：#2499

ok, I will add it.

zhaomin1423 · 2022-09-05T16:21:21Z

Please add licenses according to Dependency licenses action. you can see https://seatunnel.apache.org/docs/contribution/new-license

Thanks, I will do it.

zhaomin1423 · 2022-09-06T23:29:32Z

Please help me review it. I only add e2e in spark 3.3 for FakeSourceToConsoleIT, will add more after merge it, because the module change frequently, I fixed conflicts repeatedly. @Hisoka-X
The spark scope is changed to provided in example module, because the example don't add the release package, the compile scope need handle licenses, it's unnecessary, spark has a lot of dependencies. @ashulin

Hisoka-X · 2022-09-07T02:15:50Z

Hi, Please fix UT

zhaomin1423 · 2022-09-07T03:02:30Z

Hi, Please fix UT

The fail is related with flink. The pr don't update, can you help me rerun it?

zhaomin1423 · 2022-09-07T04:43:27Z

The fail is strange, I don't how to fix it. @Hisoka-X

Hisoka-X

Overall look good for me, @ashulin Please check source part. Thanks

Hisoka-X · 2022-09-07T08:03:38Z

How user switch spark2.4 and spark3.3 use shell command?

hailin0 · 2022-09-07T08:05:41Z

seatunnel-translation/seatunnel-translation-spark/pom.xml

    <dependencyManagement>
        <dependencies>
-            <!--spark-->
            <dependency>


move to spark-2.4 module?

Because seatunnel-translation-spark-common depend on spark, if move it, the common module need add it.

hailin0 · 2022-09-07T08:15:52Z

seatunnel-core/seatunnel-spark-starter/pom.xml

@@ -74,7 +74,7 @@

        <dependency>
            <groupId>org.apache.seatunnel</groupId>
-            <artifactId>seatunnel-translation-spark-2.4</artifactId>
+            <artifactId>seatunnel-translation-spark-common</artifactId>


How user switch seatunnel-transform-spark(2.4、3.3) list?

How user switch seatunnel-transform-spark(2.4、3.3) list?

Our release package include them, user choose the jar according to their spark client version.

How to choose? Seem like you should add jar choose logic in seatunnel-spark-starter.

Here，can put it to the dir. The e2e is using by the style.

Typically, users choose one spark version, so they only use one translation.

I prefer a more automated way, such as by judging the version of SPARK_HOME or using SPARK2 by default, you can switch versions by adding parameters to the shell script

Good idea. The arg can control to choose which plugin dir. We can do it in a new pr. How about you

zhaomin1423 · 2022-09-07T09:53:30Z

How user switch spark2.4 and spark3.3 use shell command?

According to spark client version, choose the jar. But, we can handle it later, the pr only add spark3.3 translation. There are frequent conflicts if we add more content.

Hisoka-X · 2022-09-07T09:57:25Z

How user switch spark2.4 and spark3.3 use shell command?

According to spark client version, choose the jar. But, we can handle it later, the pr only add spark3.3 translation. There are frequent conflicts if we add more content.

OK for me

zhaomin1423 · 2022-09-07T10:01:41Z

.../connector-spark-e2e-base/src/test/java/org/apache/seatunnel/e2e/spark/Spark33Container.java

+
+    @Override
+    protected String getTranslationJarTargetPath() {
+        return Paths.get(SEATUNNEL_HOME, "plugins", "translation-spark-3.3", "lib",


cc @hailin0 @Hisoka-X.

zhaomin1423 · 2022-09-09T03:06:14Z

Could you help me review it, merging is blocking, the conflicts are frequent. @ashulin @CalvinKirs

Hisoka-X · 2022-09-26T11:12:53Z

Can we support all spark e2e run both 3.3 and 2.4 without configure anything? I find we need write same code twice if we want test job both on 3.3 and 2.4. If we do this, and all e2e passed. I think we can merge this PR.

zhaomin1423 · 2022-09-28T00:55:39Z

Can we support all spark e2e run both 3.3 and 2.4 without configure anything? I find we need write same code twice if we want test job both on 3.3 and 2.4. If we do this, and all e2e passed. I think we can merge this PR.

I only add one test for 3.3, I will fix conflicts later.

dik111 · 2022-11-01T02:17:36Z

spark version :3.3.1
hive version: 3.0.0
mysql versin: 5.7
I had test mysql to hive
it throws some exception:

         client token: N/A
         diagnostics: User class threw exception: java.util.ServiceConfigurationError: org.apache.seatunnel.spark.BaseSparkSink: Provider org.apache.seatunnel.spark.hive.sink.Hive could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.loadPluginInstance(AbstractPluginDiscovery.java:128)
        at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:99)
        at org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSinks$2(SparkExecutionContext.java:95)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSinks(SparkExecutionContext.java:98)
        at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:57)
        at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
        at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
        at org.apache.seatunnel.spark.hive.sink.Hive.<init>(Hive.scala:29)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 28 more

         ApplicationMaster host: jtbihdp04.sogal.com
         ApplicationMaster RPC port: 39125
         queue: default
         start time: 1667267960414
         final status: FAILED
         tracking URL: http://jtbihdp13.sogal.com:8088/proxy/application_1667009280185_5845/
         user: suofy
22/11/01 10:00:51 ERROR Client: Application diagnostics message: User class threw exception: java.util.ServiceConfigurationError: org.apache.seatunnel.spark.BaseSparkSink: Provider org.apache.seatunnel.spark.hive.sink.Hive could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.loadPluginInstance(AbstractPluginDiscovery.java:128)
        at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:99)
        at org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSinks$2(SparkExecutionContext.java:95)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSinks(SparkExecutionContext.java:98)
        at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:57)
        at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
        at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
        at org.apache.seatunnel.spark.hive.sink.Hive.<init>(Hive.scala:29)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 28 more

Exception in thread "main" org.apache.spark.SparkException: Application application_1667009280185_5845 finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1342)
        at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

jinmu0410 · 2022-12-20T06:53:45Z

Who can help me look at this problem

jinmu0410 · 2022-12-20T06:55:34Z

Who can help me look at this problem

for test spark3 by use spark3_sink build

Hisoka-X · 2022-12-20T07:00:05Z

Who can help me look at this problem

for test spark3 by use spark3_sink build

@JinJiDeJinMu Hi, Support Spark3 not finished at now. But @nishuihanqiu have a branch for spark3 in his repository. #875 (comment)

jinmu0410 · 2022-12-20T07:14:30Z

Who can help me look at this problem

for test spark3 by use spark3_sink build

@JinJiDeJinMu Hi, Support Spark3 not finished at now. But @nishuihanqiu have a branch for spark3 in his repository. #875 (comment)

@Hisoka-X thank you

jinmu0410 · 2022-12-20T09:30:18Z

@Hisoka-X I want to join in the translation of Spark3. x based on v2, What can I do？

Hisoka-X · 2022-12-20T09:37:45Z

@Hisoka-X I want to join in the translation of Spark3. x based on v2, What can I do？

@JinJiDeJinMu Wonderful! You can continue do your work with this PR. Fix conflict and do some test, if have some problem then fix it.

zhaomin1423 mentioned this pull request Aug 30, 2022

[Feature][seatunnel-translation-spark] Support Spark-3.3 - support batch read #2329

Closed

3 tasks

zhaomin1423 marked this pull request as draft August 30, 2022 18:07

zhaomin1423 force-pushed the spark3_sink branch from 21424ad to b71d350 Compare August 30, 2022 22:20

zhaomin1423 marked this pull request as ready for review August 30, 2022 23:15

CalvinKirs added core SeaTunnel core module Spark labels Aug 31, 2022

CalvinKirs requested a review from Hisoka-X August 31, 2022 01:16

ashulin reviewed Aug 31, 2022

View reviewed changes

Hisoka-X added the Waiting for users feedback Waiting for feedback from issue/PR author label Sep 3, 2022

zhaomin1423 force-pushed the spark3_sink branch from 77412ae to 09a6cca Compare September 5, 2022 17:15

zhaomin1423 marked this pull request as draft September 5, 2022 17:26

zhaomin1423 force-pushed the spark3_sink branch 2 times, most recently from a2baef9 to a308365 Compare September 6, 2022 13:31

zhaomin1423 marked this pull request as ready for review September 6, 2022 23:24

zhaomin1423 requested review from ashulin and removed request for Hisoka-X September 6, 2022 23:30

Hisoka-X removed the Waiting for users feedback Waiting for feedback from issue/PR author label Sep 7, 2022

Hisoka-X reviewed Sep 7, 2022

View reviewed changes

hailin0 reviewed Sep 7, 2022

View reviewed changes

zhaomin1423 commented Sep 7, 2022

View reviewed changes

Hisoka-X previously approved these changes Sep 8, 2022

View reviewed changes

zhaomin1423 requested review from ashulin and hailin0 and removed request for ashulin and hailin0 September 8, 2022 14:36

zhaomin1423 added 3 commits September 9, 2022 11:46

fix

de1cff7

fix

de025c5

fix

82bfaf5

zhaomin1423 dismissed Hisoka-X’s stale review via 82bfaf5 September 9, 2022 03:47

zhaomin1423 force-pushed the spark3_sink branch from 39f5646 to 82bfaf5 Compare September 9, 2022 03:47

EricJoy2048 added the Waiting for code update label Oct 13, 2022

CalvinKirs mentioned this pull request Oct 14, 2022

[Feature] When can Spark 3. x be supported？ #3103

Closed

3 tasks

Hisoka-X mentioned this pull request Oct 18, 2022

[Bug] ClassNotFoundException: org.apache.spark.internal.Logging$class #3106

Closed

3 tasks

TyrantLucifer changed the title ~~Support seatunnel-translation-spark-3.3~~ [Feature][Connector-V2][Translation] Support spark 3.3 Nov 5, 2022

TyrantLucifer closed this Apr 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][Translation] Support spark 3.3 #2574

[Feature][Connector-V2][Translation] Support spark 3.3 #2574

zhaomin1423 commented Aug 30, 2022

ashulin left a comment •

edited

Hisoka-X commented Aug 31, 2022

zhaomin1423 commented Sep 5, 2022

zhaomin1423 commented Sep 5, 2022

zhaomin1423 commented Sep 6, 2022 •

edited

Hisoka-X commented Sep 7, 2022

zhaomin1423 commented Sep 7, 2022

zhaomin1423 commented Sep 7, 2022 •

edited

Hisoka-X left a comment

Hisoka-X commented Sep 7, 2022

hailin0 Sep 7, 2022

zhaomin1423 Sep 7, 2022

hailin0 Sep 7, 2022

zhaomin1423 Sep 7, 2022

Hisoka-X Sep 7, 2022

zhaomin1423 Sep 7, 2022

zhaomin1423 Sep 7, 2022

Hisoka-X Sep 7, 2022

zhaomin1423 Sep 7, 2022

Hisoka-X Sep 7, 2022

zhaomin1423 commented Sep 7, 2022

Hisoka-X commented Sep 7, 2022

zhaomin1423 Sep 7, 2022

zhaomin1423 commented Sep 9, 2022

Hisoka-X commented Sep 26, 2022

zhaomin1423 commented Sep 28, 2022

dik111 commented Nov 1, 2022

jinmu0410 commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

Hisoka-X commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

Hisoka-X commented Dec 20, 2022

[Feature][Connector-V2][Translation] Support spark 3.3 #2574

[Feature][Connector-V2][Translation] Support spark 3.3 #2574

Conversation

zhaomin1423 commented Aug 30, 2022

Purpose of this pull request

Check list

ashulin left a comment • edited

Choose a reason for hiding this comment

Hisoka-X commented Aug 31, 2022

zhaomin1423 commented Sep 5, 2022

zhaomin1423 commented Sep 5, 2022

zhaomin1423 commented Sep 6, 2022 • edited

Hisoka-X commented Sep 7, 2022

zhaomin1423 commented Sep 7, 2022

zhaomin1423 commented Sep 7, 2022 • edited

Hisoka-X left a comment

Choose a reason for hiding this comment

Hisoka-X commented Sep 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhaomin1423 commented Sep 7, 2022

Hisoka-X commented Sep 7, 2022

Choose a reason for hiding this comment

zhaomin1423 commented Sep 9, 2022

Hisoka-X commented Sep 26, 2022

zhaomin1423 commented Sep 28, 2022

dik111 commented Nov 1, 2022

jinmu0410 commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

Hisoka-X commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

jinmu0410 commented Dec 20, 2022

Hisoka-X commented Dec 20, 2022

ashulin left a comment •

edited

zhaomin1423 commented Sep 6, 2022 •

edited

zhaomin1423 commented Sep 7, 2022 •

edited