Some wrong with mxnet on spark: params.jars = jars.split(",|:")

[liuzx32]

The method setExecutorJars in org.apache.mxnet.spark.MXNet.scala has 'params.jars = jars.split(",|:")' sentence. If the jars from s3://path or hdfs://path, there will be wrong jar path, for example s3 or hdfs or others. Perhaps 'params.jars = jars.split(",")' is recommended.

[anirudhacharya]

@nswamy @lanking520 could one of you take a look?

[lanking520]

@yzhliu @nswamy @@andrewfayres .I think we need to build regex to identify these different formats, will start a PR and try if I can solve that. @nswamy can you label this as Scala and bug in here?

[yzhliu]

An alternative is to allow users to use quotation mark. string in quotations should never be split: "s3://path":"hdfs://path"

[lanking520]

Hi @liuzx32 , the jars will be used in this way:

jars.map(jar => SparkFiles.get(new File(jar).getName)).mkString(":")

It seemed we cannot directly place S3 or HDFS paths in here. So It's not necessary to initiate this change.

[nswamy]

Not an issue at the moment and We will add a Feature Request to support jars from S3 and HDFS.

[liuzx32]

@yzhliu @lanking520 Mxnet on spark should be for yarn cluster mode not only yarn local mode. If every machine have the same configure of mxnet, why we need yarn or spark?

[liuzx32]

@lanking520 Perhaps the problem is here:
jars.map(jar => SparkFiles.get(new File(jar).getName)).mkString(":")
With the sentence how to run on yarn?