could not connect to Doris in Spark when in a cluster mode

[no7dw]

dengwei@cluster-6a52-m:/usr/lib/spark$ bin/spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/05/10 10:30:08 INFO org.apache.spark.SparkEnv: Registering MapOutputTracker 22/05/10 10:30:08 INFO org.apache.spark.SparkEnv: Registering BlockManagerMaster 22/05/10 10:30:08 INFO org.apache.spark.SparkEnv: Registering BlockManagerMasterHeartbeat 22/05/10 10:30:09 INFO org.apache.spark.SparkEnv: Registering OutputCommitCoordinator Spark context Web UI available at http://cluster-6a52-m.us-central1-a.c.xed-project-237404.internal:34191 Spark context available as 'sc' (master = yarn, app id = application_1652096761512_0003). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ / / '/
// .__/_,// //_\ version 3.1.2
//

Using Scala version 2.12.14 (OpenJDK 64-Bit Server VM, Java 1.8.0_322)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.doris.spark._
import org.apache.doris.spark._

scala> val dorisSparkRDD = sc.dorisRDD(
| tableIdentifier = Some("example_db.ttt"),
| cfg = Some(Map(
| "doris.fenodes" -> "test_host:8031",
| "doris.request.auth.user" -> "test",
| "doris.request.auth.password" -> "test"
| ))
| )
dorisSparkRDD: org.apache.spark.rdd.RDD[AnyRef] = ScalaDorisRDD[0] at RDD at AbstractDorisRDD.scala:32

scala> dorisSparkRDD.collect()`

22/05/10 10:30:46 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 2) (cluster-6a52-w-0.us-central1-a.c.xed-project-237404.internal executor 1): java.lang.ClassNotFoundException: org.apache.doris.spark.rdd.DorisPartition
at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:124)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

currently env:
spark 3.1.2
doris-spark-connector : doris-spark-connector-3.1.2-2.12-1.0.0.jar
os: Debian 10
on google cloud , dataproc

the above code works fine in standalone mode.
but run with error in cluster

the connector was built in docker support by official ( centos) 1.0.0-rc03
this blog has similar case , but the details error is different.
the blog said it was because spark 2.x supported rather than 3.x in 2021.
but the official link here supported 3.2.x already .

thanks in advance for any clue ~

[no7dw]

after help from community, this article
Yarn集群模式运行的Spark，则将此文件放入预部署包中。
预部署包 or so called pre-deployment package

the package folder which located according to config file : $SPARK_HOME/conf/spark-defaults.conf
spark.yarn.jars=local:/usr/lib/spark/jars/*

by appending the hdfs file path , and retry will work
spark.yarn.jars=local:/usr/lib/spark/jars/*,hdfs:///spark-jars/doris-spark-connector-3.1.2-2.12-1.0.0.jar

$hdfs dfs -mkdir /spark-jars/
$hdfs dfs -put /usr/lib/spark/jars/doris-spark-connector-3.1.2-2.12-1.0.0.jar /spark-jars/
$ hdfs dfs -ls /spark-jars/
Found 1 items
-rw-r--r--   2 dengwei hadoop    7280507 2022-05-11 16:09 /spark-jars/doris-spark-connector-3.1.2-2.12-1.0.0.jar

#