[Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found

[zhouhongyu888]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I hava deployed ds by docker-compose and mount the spark-client on localhost，when I submit a spark-task(the example task of sparkPi), the error below happends:
[INFO] 2022-07-13 08:30:49.424 +0000 [taskAppId=TASK-20220713-6178360895936_1-48-48] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[63] - -> /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: 4: /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: source: not found
/tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: 5: /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: /bin/spark-submit: not found

What you expected to happen

the task can run successfully

How to reproduce

1、deployed dolphinscheduler by docker-compose
2、mount the directory /opt/soft on localhost

Anything else

No response

Version

3.0.0-beta-2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[SbloodyS]

It seems you lack of spark environment.

[zhouhongyu888]

It seems you lack of spark environment.

This is a bug，when execute the command, it seems to use /bin/spark-submit, not $SPARK_HOME2/bin/spark-submit。so I copy the scripts in the directory of $SPARK_HOME2/bin to the directory of /bin，the bug disappears

[SbloodyS]

/tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: source: not found

The reason for the failure is that the source environment variable failed. Please check it.

[itzhang89]

I have the same issue when use the 3.0.0-release version. The log as brew.

[LOG-PATH]: /opt/dolphinscheduler/logs/20220812/6508614388448_7-10-14.log, [HOST]:  Host{address='my-do-worker-1.my-do-worker-headless:1234', ip='my-do-worker-1.my-do-worker-headless', port=1234}
[INFO] 2022-08-12 19:12:36.086 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[69] - spark task params {"localParams":[],"rawScript":"","resourceList":[],"programType":"SCALA","mainClass":"org.apache.spark.examples.SparkPi","mainJar":{"id":2,"resourceName":"/spark-examples_2.12-3.2.1.jar","res":"spark-examples_2.12-3.2.1.jar"},"deployMode":"client","sparkVersion":"SPARK2","driverCores":1,"driverMemory":"512M","numExecutors":2,"executorMemory":"2G","executorCores":2}
[INFO] 2022-08-12 19:12:36.091 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[130] - spark task command: ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[85] - tenantCode user:telent01, task dir:10_14
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[90] - create command file:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[116] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
[INFO] 2022-08-12 19:12:36.096 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[290] - task run command: sudo -u telent01 sh /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
[INFO] 2022-08-12 19:12:36.097 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[181] - process start, process id is: 7776
[INFO] 2022-08-12 19:12:36.102 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[205] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14, processId:7776 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-08-12 19:12:37.098 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[63] -  -> /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 4: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: source: not found
	/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 5: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: /bin/spark-submit: not found
[INFO] 2022-08-12 19:12:37.099 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[57] - FINALIZE_SESSION

The default script compiler is sh, not the bash. and the #!/bin/sh not have the source command default. If you want to make the environment take effect. you can overwrite the environment on the workflow.

[ruanwenjun]

Can we use #!/bin/bash to replace #!/bin/sh? @SbloodyS @caishunfeng