Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Spark] Can't get config file in spark standalone cluster. #1771

Open
2 of 3 tasks
Hisoka-X opened this issue Apr 28, 2022 · 0 comments
Open
2 of 3 tasks

[Bug] [Spark] Can't get config file in spark standalone cluster. #1771

Hisoka-X opened this issue Apr 28, 2022 · 0 comments

Comments

@Hisoka-X
Copy link
Member

Hisoka-X commented Apr 28, 2022

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

When use spark standalone cluster to submit job, and use cluster mode. Driver can't read config file. Maybe we should use SparkFile api to get file path. @whb-bigdata I find you fix the same issue on #1747 . Do you have any suggestion?

SeaTunnel Version

dev

SeaTunnel Config

-

Running Command

./bin/start-seatunnel-spark.sh --config config/spark.batch.conf -m spark://localhost:7077 -e cluster

Error Exception

Launch Command: "/usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java" "-cp" "/home/hadoop/spark/conf/:/home/hadoop/spark/jars/*:/home/hadoop/hadoop/etc/hadoop/" "-Xmx1024M" "-Dspark.executor.memory=512m" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.executor.instances=2" "-Dspark.files=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/plugins.tar.gz,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/config/spark.batch.conf" "-Dspark.master=spark://172.31.0.76:7077" "-Dspark.jars=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-fake-2.1.1-SNAPSHOT.jar,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-console-2.1.1-SNAPSHOT.jar,file:/home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/lib/seatunnel-core-spark.jar" "-Dspark.app.name=SeaTunnelGhy" "-Dspark.executor.cores=1" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@172.31.0.76:41397" "/home/hadoop/spark/work/driver-20220428082338-0000/seatunnel-core-spark.jar" "org.apache.seatunnel.SeatunnelSpark" "--config" "config/spark.batch.conf" "-m" "spark://172.31.0.76:7077" "-e" "cluster"
========================================

22/04/28 08:23:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/04/28 08:23:40 INFO util.Utils: Successfully started service 'Driver' on port 44727.
22/04/28 08:23:40 INFO worker.DriverWrapper: Driver address: 172.31.0.76:44727
22/04/28 08:23:40 INFO worker.WorkerWatcher: Connecting to worker spark://Worker@172.31.0.76:41397
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root
22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to:
22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/04/28 08:23:40 INFO client.TransportClientFactory: Successfully created connection to /172.31.0.76:41397 after 30 ms (0 ms spent in bootstraps)
22/04/28 08:23:40 INFO worker.WorkerWatcher: Successfully connected to spark://Worker@172.31.0.76:41397
22/04/28 08:23:41 INFO config.ConfigBuilder: Loading config file: spark.batch.conf
22/04/28 08:23:41 INFO config.ConfigBuilder: parsed config file: {}

22/04/28 08:23:41 ERROR seatunnel.Seatunnel:

===============================================================================


22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Fatal Error,

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Reason:No configuration setting found for key 'env'

22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
	at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
	at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38)
	at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
	at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)

22/04/28 08:23:41 ERROR seatunnel.Seatunnel:
===============================================================================



Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
	at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
	at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
	at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46)
	at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38)
	at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
	at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
	... 6 more
22/04/28 08:23:41 INFO util.ShutdownHookManager: Shutdown hook called
22/04/28 08:23:41 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b3013229-90c1-43d5-999b-adff6ef7652e

Flink or Spark Version

spark 2.4.0

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants