[Bug] [Spark] Can't get config file in spark standalone cluster.

[Hisoka-X]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When use spark standalone cluster to submit job, and use cluster mode. Driver can't read config file. Maybe we should use SparkFile api to get file path. @whb-bigdata I find you fix the same issue on #1747 . Do you have any suggestion?

SeaTunnel Version

dev

SeaTunnel Config

-

Running Command

./bin/start-seatunnel-spark.sh --config config/spark.batch.conf -m spark://localhost:7077 -e cluster

Error Exception

Launch Command: "/usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java" "-cp" "/home/hadoop/spark/conf/:/home/hadoop/spark/jars/*:/home/hadoop/hadoop/etc/hadoop/" "-Xmx1024M" "-Dspark.executor.memory=512m" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.executor.instances=2" "-Dspark.files=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/plugins.tar.gz,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/config/spark.batch.conf" "-Dspark.master=spark://172.31.0.76:7077" "-Dspark.jars=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-fake-2.1.1-SNAPSHOT.jar,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-console-2.1.1-SNAPSHOT.jar,file:/home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/lib/seatunnel-core-spark.jar" "-Dspark.app.name=SeaTunnelGhy" "-Dspark.executor.cores=1" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@172.31.0.76:41397" "/home/hadoop/spark/work/driver-20220428082338-0000/seatunnel-core-spark.jar" "org.apache.seatunnel.SeatunnelSpark" "--config" "config/spark.batch.conf" "-m" "spark://172.31.0.76:7077" "-e" "cluster"
========================================

22/04/28 08:23:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/04/28 08:23:40 INFO util.Utils: Successfully started service 'Driver' on port 44727.
22/04/28 08:23:40 INFO worker.DriverWrapper: Driver address: 172.31.0.76:44727
22/04/28 08:23:40 INFO worker.WorkerWatcher: Connecting to worker spark://Worker@172.31.0.76:41397
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/04/28 08:23:40 INFO client.TransportClientFactory: Successfully created connection to /172.31.0.76:41397 after 30 ms (0 ms spent in bootstraps)
22/04/28 08:23:40 INFO worker.WorkerWatcher: Successfully connected to spark://Worker@172.31.0.76:41397
22/04/28 08:23:41 INFO config.ConfigBuilder: Loading config file: spark.batch.conf
22/04/28 08:23:41 INFO config.ConfigBuilder: parsed config file: {}

22/04/28 08:23:41 ERROR seatunnel.Seatunnel:

===============================================================================


22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Fatal Error,

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Reason:No configuration setting found for key 'env'

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
	at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
	at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38)
	at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
	at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)

22/04/28 08:23:41 ERROR seatunnel.Seatunnel:
===============================================================================



Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
	at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
	at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38)
	at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
	at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
	... 6 more
22/04/28 08:23:41 INFO util.ShutdownHookManager: Shutdown hook called
22/04/28 08:23:41 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b3013229-90c1-43d5-999b-adff6ef7652e

Flink or Spark Version

spark 2.4.0

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct