Bug in run_spark?

[sciaba]

I noticed that in

https://github.com/dmwm/CMSSpark/blob/master/bin/run_spark#L54

the Spark version is identified as 2 only if spark-submit is in the path, which is certainly not the case on lxplus7. So, running from lxplus7 one gets "Using spark 1.X" in the output, which is typically wrong.
From lxplus7, $jars ends up being

--jars /afs/cern.ch/user/v/valya/public/spark/spark-csv-assembly-1.4.0.jar,/afs/cern.ch/user/v/valya/public/spark/avro-mapred-1.7.6-cdh5.7.6.jar,/afs/cern.ch/user/v/valya/public/spark/spark-examples-1.6.0-cdh5.15.1-hadoop2.6.0-cdh5.15.1.jar

which looks suspicious because the spark-examples version is for 1.6.0. The version for Spark 2 is never found because it's always looked for under /usr/hdp/.
I don't know if this is a real problem, at least it doesn't seem to affect my CMSSpark code.

By the way, in the official instructions for running Spark from lxplus7, /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-centos7-gcc62-opt had been changed to /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-centos7-gcc7-opt.

Cheers,
Andrea

[vkuznet]

Andrea, I committed the change in this commit: 33c99b8..4dbddef Feel free to download recent version from master head and try it out. Best, Valentin.

…

On 0, Andrea Sciaba ***@***.***> wrote: I noticed that in https://github.com/dmwm/CMSSpark/blob/master/bin/run_spark#L54 the Spark version is identified as 2 only if spark-submit is in the path, which is certainly not the case on lxplus7. So, running from lxplus7 one gets "Using spark 1.X" in the output, which is typically wrong. From lxplus7, $jars ends up being --jars /afs/cern.ch/user/v/valya/public/spark/spark-csv-assembly-1.4.0.jar,/afs/cern.ch/user/v/valya/public/spark/avro-mapred-1.7.6-cdh5.7.6.jar,/afs/cern.ch/user/v/valya/public/spark/spark-examples-1.6.0-cdh5.15.1-hadoop2.6.0-cdh5.15.1.jar which looks suspicious because the spark-examples version is for 1.6.0. The version for Spark 2 is never found because it's always looked for under /usr/hdp/. I don't know if this is a real problem, at least it doesn't seem to affect my CMSSpark code. By the way, in the official instructions for running Spark from lxplus7, /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-centos7-gcc62-opt had been changed to /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-centos7-gcc7-opt. Cheers, Andrea -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #26